An Efficient, Sparsity-Preserving, Online Algorithm for Low-RankApproximation

David Anderson * 1 Ming Gu * 1

AbstractLow-rank matrix approximation is a fundamentaltool in data analysis for processing large datasets,reducing noise, and finding important signals.In this work, we present a novel truncated LUfactorization called Spectrum-Revealing LU(SRLU) for effective low-rank matrix approxi-mation, and develop a fast algorithm to computean SRLU factorization. We provide both matrixand singular value approximation error boundsfor the SRLU approximation computed by ouralgorithm. Our analysis suggests that SRLU iscompetitive with the best low-rank matrix ap-proximation methods, deterministic or random-ized, in both computational complexity and ap-proximation quality. Numeric experiments illus-trate that SRLU preserves sparsity, highlights im-portant data features and variables, can be effi-ciently updated, and calculates data approxima-tions nearly as accurately as possible. To the bestof our knowledge this is the first practical variantof the LU factorization for effective and efficientlow-rank matrix approximation.

1. IntroductionLow-rank approximation is an essential data processingtechnique for understanding large or noisy data in diverseareas including data compression, image and pattern recog-nition, signal processing, compressed sensing, latent se-mantic indexing, anomaly detection, and recommendationsystems. Recent machine learning applications includetraining neural networks (Jaderberg et al., 2014; Kirk-patrick et al., 2017), second order online learning (Luoet al., 2016), representation learning (Wang et al., 2016),and reinforcement learning (Ghavamzadeh et al., 2010).

*Equal contribution 1University of California, Berke-ley. Correspondence to: David Anderson <davidander-son@berkeley.edu>, Ming Gu <mgu@berkeley.edu>.

Proceedings of the 34 th International Conference on MachineLearning, Sydney, Australia, PMLR 70, 2017. Copyright 2017by the author(s).

Additionally, a recent trend in machine learning is to in-clude an approximation of second order information forbetter accuracy and faster convergence (Krummenacheret al., 2016).

In this work, we introduce a novel low-rank approximationalgorithm called Spectrum-Revealing LU (SRLU) that canbe efficiently computed and updated. Furthermore, SRLUpreserves sparsity and can identify important data variablesand observations. Our algorithm works on any data matrix,and achieves an approximation accuracy that only differsfrom the accuracy of the best approximation possible forany given rank by a constant factor.1

The major innovation in SRLU is the efficient calculationof a truncated LU factorization of the form

Π1AΠT2 =

( k m − k

k L11

m − k L21 In−k

) ( k n − k

U11 U12

S

)≈

(L11

L21

)(U11 U12

)def= LU,

where Π1 and Π2 are judiciously chosen permutation ma-trices. The LU factorization is unstable, and in practiceis implemented by pivoting (interchanging) rows duringfactorization, i.e. choosing permutation matrix Π1. Forthe truncated LU factorization to have any significance,nevertheless, complete pivoting (interchanging rows andcolumns) is necessary to guarantee that the factors L andU are well-defined and that their product accurately repre-sents the original data. Previously, complete pivoting wasimpractical as a matrix factorization technique because itrequires accessing the entire data matrix at every iteration,but SRLU efficiently achieves complete pivoting throughrandomization and includes a deterministic follow-up pro-cedure to ensure a hight quality low-rank matrix approxi-mation, as supported by rigorous theory and numeric ex-periments.

1The truncated SVD is known to provide the best low-rankmatrix approximation, but it is rarely used for large scale practicaldata analysis. See a brief discussion of the SVD in supplementalmaterial.

An Efficient Algorithm for Low-Rank Approximation

1.1. Background on the LU factorization

Algorithm 1 presents a basic implementation of the LU fac-torization, where the result is stored in place such that theupper triangular part of A becomes U and the strictly lowertriangular part becomes the strictly lower part of L, withthe diagonal of L implicitly known to contain all ones. LUwith partial pivoting finds the largest entry in the ith columnfrom row i tom and pivots the row with that entry to the ith

row. LU with complete pivoting finds the largest entry inthe submatrix Ai+1:m,i+1:n and pivots that entry to Ai,i. Itis generally known and accepted that partial pivoting is suf-ficient for general, real-world data matrices in the contextof linear equation solving.

Algorithm 1 The LU factorization

1: Inputs: Data matrix A ∈ Rm×n2: for i = 1, 2, · · · ,min(m,n) do3: Perform row and/or column pivots4: for k = i+ 1, · · · ,m do5: Ak,i = Ak,i/Ai,i

6: end for7: Ai+1:m,i+1:n −= Ai+1:m,1:i ·A1:i,i+1:n

8: end for

Algorithm 2 Crout LU

1: Inputs: Data matrix A ∈ Rm×n, block size b2: for j = 0, b, 2b, · · · ,min(m,n)/b− 1 do3: Perform column pivots4: Aj+1:m,j+1:j+b− =5: Aj+1:m,1:j ·A1:j,j+1:j+b.6: Apply Algorithm 1 on Aj+1:m,j+1:j+b

7: Apply the row pivots to other columns of A8: Aj+1:j+b,j+b+1:n −=9: Aj+1:j+b,1:j ·A1:j,j+b+1:n

10: end for

Line 7 of Algorithm 1 is known as the Schur update. Givena sparse input, this is the only step of the LU factorizationthat causes fill. As the algorithm progresses, fill will com-pound and may become dense, but the LU factorization,and truncated LU in particular, generally preserves some,if not most, of the sparsity of a sparse input. A numericillustration is presented below.

There are many variations of the LU factorization. In Al-gorithm 2 the Crout version of LU is presented in blockform. The column pivoting entails selecting the next bcolumns so that the in-place LU step is performed on anon-singular matrix (provided the remaining entries are notall zero). Note that the matrix multiplication steps are thebottleneck of this algorithm, requiring O(mnb) operationseach in general.

The LU factorization has been studied extensively sincelong before the invention of computers, with notable re-sults from many mathematicians, including Gauss, Turing,and Wilkinson. Current research on LU factorizations in-cludes communication-avoiding implementations, such astournament pivoting (Khabou et al., 2013), sparse imple-mentations (Grigori et al., 2007), and new computationof preconditioners (Chow & Patel, 2015). A randomizedapproach to efficiently compute the LU factorization withcomplete pivoting recently appeared in (Melgaard & Gu,2015). These results are all in the context of linear equationsolving, either directly or indirectly through an incompletefactorization used to precondition an iterative method. Thiswork repurposes the LU factorization to create a novel effi-cient and effective low-rank approximation algorithm usingmodern randomization technology.

2. Previous Work2.1. Low-Rank Matrix Approximation (LRMA)

Previous work on low-rank data approximation includesthe Interpolative Decomposition (ID) (Cheng et al., 2005),the truncated QR with column pivoting factorization (Gu &Eisenstat, 1996), and other deterministic column selectionalgorithms, such as in (Batson et al., 2012).

Randomized algorithms have grown in popularity in recentyears because of their ability to efficiently process largedata matrices and because they can be supported with rig-orous theory. Randomized low-rank approximation algo-rithms generally fall into one of two categories: samplingalgorithms and black box algorithms. Sampling algorithmsform data approximations from a random selection of rowsand/or columns of the data. Examples include (Desh-pande et al., 2006; Deshpande & Vempala, 2006; Friezeet al., 2004; Mahoney & Drineas, 2009). (Drineas et al.,2008) showed that for a given approximate rank k, a ran-domly drawn subset C of c = O

(k log(k)ε−2 log (1/δ)

)columns of the data, a randomly drawn subset R ofr = O

(c log(c)ε−2 log (1/δ)

)rows of the data, and set-

ting U = C†AR†, then the matrix approximation error‖A−CUR‖F is at most a factor of 1+ ε from the optimalrank k approximation with probability at least 1− δ. Blackbox algorithms typically approximate a data matrix in theform

A ≈ QTQA,

where Q is an orthonormal basis of the random projection(usually using SVD, QR, or ID). The result of (Johnson &Lindenstrauss, 1984) provided the theoretical groundworkfor these algorithms, which have been extensively studied(Clarkson & Woodruff, 2012; Halko et al., 2011; Martins-son et al., 2006; Papadimitriou et al., 2000; Sarlos, 2006;Woolfe et al., 2008; Liberty et al., 2007; Gu, 2015). Note

An Efficient Algorithm for Low-Rank Approximation

that the projection of an m-by-n data matrix is of size m-by-`, for some oversampling parameter ` ≥ k, and k isthe target rank. Thus the computational challenge is theorthogonalization of the projection (the random projectioncan be applied quickly, as described in these works). A pre-vious result on randomized LU factorizations for low-rankapproximation was presented in (Aizenbud et al., 2016),but is uncompetitive in terms of theoretical results and com-putational performance with the work presented here.

For both sampling and black box algorithms the tuning pa-rameter ε cannot be arbitrarily small, as the methods be-come meaningless if the number of rows and columns sam-pled (in the case of sampling algorithms) or the size of therandom projection (in the case of black box algorithms)surpasses the size of the data. A common practice is ε ≈ 1

2 .

2.2. Guaranteeing Quality

Rank-revealing algorithms (Chan, 1987) are LRMA algo-rithms that guarantee the approximation is of high qualityby also capturing the rank of the data within a tolerance(see supplementary materials for definitions). These meth-ods, nevertheless, attempt to build an important submatrixof the data, and do not directly compute a low-rank ap-proximation. Furthermore, they do not attempt to captureall positive singular values of the data. (Miranian & Gu,2003) introduced a new type of high-quality LRMA algo-rithms that can capture all singular values of a data matrixwithin a tolerance, but requires extra computation to boundapproximations of the left and right null spaces of the datamatrix. Rank-revealing algorithms in general are designedaround a definition that is not specifically appropriate forLRMA.

A key advancement of this work is a new definition of highquality low-rank approximation:

Definition 1 A rank-k truncated LU factorization isspectrum-revealing if∥∥∥A− LU

∥∥∥2≤ q1(k,m, n)σk+1 (A)

and

σj

(LU)≥ σj (A)

q2(k,m, n)

for 1 ≤ j ≤ k and q1(k,m, n) and q2(k,m, n) arebounded by a low degree polynomial in k, m, and n.

Definition 1 has precisely what we desire in an LRMA,and no additional requirements. The constants, q1(k,m, n)and q2(k,m, n) are at least 1 for any rank-k approximationby (Eckart & Young, 1936). This work shows theoreticallyand numerically that our algorithm, SRLU, is spectrum-revealing in that it always finds such q1 and q2, often withq1, q2 = O(1) in practice.

Algorithm 3 TRLUCP

1: Inputs: Data matrix A ∈ Rm×n, target rank k, blocksize b, oversampling parameter p ≥ b, random Gaus-sian matrix Ω ∈ Rp×m, L and U are initially 0 matri-ces

2: Calculate random projection R = ΩA3: for j = 0, b, 2b, · · · , k − b do4: Perform column selection algorithm on R and swap

columns of A5: Update block column of L6: Perform block LU with partial row pivoting and

swap rows of A7: Update block row of U8: Update R9: end for

2.3. Low-Rank and Other Approximations in MachineLearning

Low-rank and other approximation algorithms have ap-peared recently in a variety of machine learning applica-tions. In (Krummenacher et al., 2016), randomized low-rank approximation is applied directly to the adaptive opti-mization algorithm ADAGRAD to incorporate variable de-pendence during optimization to approximate the full ma-trix version of ADAGRAD with a significantly reducedcomputational complexity. In (Kirkpatrick et al., 2017), adiagonal approximation of the posterior distribution of pre-vious data is utilized to alleviate catastrophic forgetting.

3. Main Contribution: Spectrum-RevealingLU (SRLU)

Our algorithm for computing SRLU is composed of twosubroutines: partially factoring the data matrix with ran-domized complete pivoting (TRLUCP) and performingswaps to improve the quality of the approximation (SRP).The first provides an efficient algorithm for computing atruncated LU factorization, whereas the second ensures theresulting approximation is provably reliable.

3.1. Truncated Randomized LU with CompletePivoting (TRLUCP)

Intuitively, TRLUCP performs deterministic LU with par-tial row pivoting for some initial data with permutedcolumns. TRLUCP uses a random projection of the Schurcomplement to cheaply find and move forward columnsthat are more likely to be representative of the data. To ac-complish this, Algorithm 3 performs an iteration of blockLU factorization in a careful order that resembles Crout LUreduction. The ordering is reasoned as follows: LU withpartial row pivoting cannot be performed until the needed

An Efficient Algorithm for Low-Rank Approximation

columns are selected, and so column selection must firstoccur at each iteration. Once a block column is selected, apartial Schur update must be performed on that block col-umn before proceeding. At this point, an iteration of blockLU with partial row pivoting can be performed on the cur-rent block. Once the row pivoting is performed, a partialSchur update of the block of pivoted rows of U can be per-formed, which completes the factorization up to rank j+ b.Finally, the projection matrix R can be cheaply updated toprepare for the next iteration. Note that any column se-lection method may be used when picking column pivotsfrom R, such as QR with column pivoting, LU with rowpivoting, or even this algorithm can again be run on thesubproblem of column selection of R. The flop count ofTRLUCP is dominated by the three matrix multiplicationsteps (lines 2, 5, and 7). The total number of flops is

F TRLUCP = 2pmn+ (m+ n)k2 +O (k(m+ n)) .

Note the transparent constants, and, because matrix mul-tiplication is the bottleneck, this algorithm can be imple-mented efficiently in terms of both computation as wellas memory usage. Because the output of TRLUCP isonly written once, the total number of memory writes is(m + n − k)k. Minimizing the number of data writes byonly writing data once significantly improves efficiency be-cause writing data is typically one of the slowest compu-tational operations. Also worth consideration is the sim-plicity of the LU decomposition, which only involves threetypes of operations: matrix multiply, scaling, and pivoting.By contrast, state-of-the-art calculation of both the full andtruncated SVD requires a more complex process of bidiag-onalization. The projection R can be updated efficientlyto become a random projection of the Schur complementfor the next iteration. This calculation involves the currentprogress of the LU factorization and the random matrix Ω,and is described in detail in the appendix.

3.2. Spectrum-Revealing Pivoting (SRP)

TRLUCP produces high-quality data approximations foralmost all data matrices, despite the lack of theoreticalguarantees, but can miss important rows or columns of thedata. Next, we develop an efficient variant of the existingrank-revealing LU algorithms (Gu & Eisenstat, 1996; Mira-nian & Gu, 2003) to rapidly detect and, if necessary, correctany possible matrix approximation failures of TRLUCP.

Intuitively, the quality of the factorization can be tested bysearching for the next choice of pivot in the Schur com-plement if the factorization continued and determining ifthe addition of that element would significantly improvethe approximation quality. If so, then the row and columnwith this element should be included in the approximationand another row and column should be excluded to main-tain rank. Because TRLUCP does not provide an updated

Schur complement, the largest element in the Schur com-plement can be approximated by finding the column of Rwith largest norm, performing a Schur update of that col-umn, and then picking the largest element in that column.Let α be this element, and, without loss of generality, as-sume it is the first entry of the Schur complement. Denote:

Π1AΠT2 =

L11

`T 1L31 I

U11 u U13

α sT12s21 S22

. (1)

Next, we must find the row and column that should be re-placed if the row and column containing α are important.Note that the smallest entry of L11U11 may still lie in animportant row and column, and so the largest element ofthe inverse should be examined instead. Thus we proposedefining

A11def=

(L11

`T 1

)(U11 u

α

)and testing

‖A−111 ‖max ≤f

|α|(2)

for a tolerance parameter f > 1 that provides a control ofaccuracy versus the number of swaps needed. Should thetest fail, the row and column containing α are swapped withthe row and column containing the largest element in A

−111 .

Note that this element may occur in the last row or last col-umn of A

−111 , indicating only a column swap or row swap

respectively is needed. When the swaps are performed,the factorization must be updated to maintain truncated LUform. We have developed a variant of the LU updating al-gorithm of (Gondzio, 2007) to efficiently update the SRLUfactorization.

SRP can be implemented efficiently: each swap requiresat most O (k(m+ n)) operations, and ‖A−111 ‖max can bequickly and reliably estimated using (Higham & Relton,2015). An argument similar to that used in (Miranian &Gu, 2003) shows that each swap will increase

∣∣det (A11

)∣∣by a factor at least f , hence will never repeat. At termina-tion, SRP will ensure a partial LU factorization of the form(1) that satisfies condition (2). We will discuss spectrum-revealing properties of this factorization in Section 4.2.

It is possible to derive theoretical upper bounds on theworst number of swaps necessary in SRP, but in practice,this number is zero for most matrices, and does not exceed3− 5 in the most pathological data matrix of dimension atmost 1000 we can contrive.

SRLU can be used effectively to approximate second or-der information in machine learning. SRLU can be usedas a modification to ADAGRAD in a manner similar to the

An Efficient Algorithm for Low-Rank Approximation

Algorithm 4 Spectrum-Revealing Pivoting (SRP)

1: Input: Truncated LU factorization A ≈ LU, toler-ance f > 1

2: while ‖A−111 ‖max >f|α| do

3: Set α to be the largest element in S (or find an ap-proximate α using R)

4: Swap row and column containing α with row andcolumn of largest element in A

−111

5: Update truncated LU factorization6: end while

low-rank approximation method in (Krummenacher et al.,2016). Applying the initialization technique in this work,SRLU would likely provide an efficient and accurate adap-tive stochastic optimization algorithm. SRLU can also be-come a full-rank approximation (low-rank plus diagonal)by adding a diagonal approximation of the Schur comple-ment. Such an approximation could be appropriate for im-proving memory in artificial intelligence, such as in (Kirk-patrick et al., 2017). SRLU is also a freestanding compres-sion algorithm.

3.3. The CUR Decomposition with LU

A natural extension of truncated LU factorizations isa CUR-type decomposition for increased accuracy (Ma-honey & Drineas, 2009):

Π1AΠT2 ≈ L

(L†AU†

)U

def= LMU.

As with standard CUR, the factors L and U retain (muchof) the sparsity of the original data, while M is a small,k-by-k matrix. The CUR decomposition can improve theaccuracy of an SRLU with minimal extra needed memory.Extra computational time, nevertheless, is needed to calcu-late M. A more efficient, approximate CUR decompositioncan be obtained by replacing A with a high quality approx-imation (such as an SRLU factorization of high rank) in thecalculation of M.

3.4. The Online SRLU Factorization

Given a factored data matrix A ∈ Rm×n and new observa-

tions BΠT2 =

( k m − k

B1 B2

)∈ Rs×m, an augmented LU

decomposition takes the form(Π1AΠT

2

BΠT2

)=

L11

L21 IL31 I

U11 U12

SSnew

,

where L31 = B1U−111 and Snew = B2 −B1U

−111 U12. An

SRLU factorization can then be obtained by simply per-forming correcting swaps. For a rank-1 update, at most 1

swap is expected (although examples can be constructedthat require more than one swap), which requires at mostO (k (m+ n)) flops. By contrast, the URV decomposi-tion of (Stewart, 1992) is O

(n2), while SVD updating

requires O((m+ n) min2 (m,n)

)operations in general,

or O((m+ n) min (m,n) log2

2 ε)

for a numerical approx-imation with the fast multipole method.

4. Theoretical Results for SRLUFactorizations

4.1. Analysis of General Truncated LU Decompositions

Theorem 1 Let (·)s denote the rank-s truncated SVD fors ≤ k m,n. Then for any truncated LU factorizationwith Schur complement S:

‖Π1AΠT2 − LU‖ = ‖S‖

for any norm, and

‖Π1AΠT2 −

(LU)s‖2 ≤ 2‖S‖2 + σs+1 (A) .

Theorem 2 For a general rank-k truncated LU decompo-sition, we have for all 1 ≤ j ≤ k,

σj (A) ≤ σj(LU)1 +

1 +‖S‖2

σk

(LU) ‖S‖2σj (A)

.

Theorem 3 CUR Error Bounds.

‖Π1AΠT2 − LMU‖2 ≤ 2‖S‖2

and

‖Π1AΠT2 − LMU‖F ≤ ‖S‖F .

Theorem 1 simply concludes that the approximation is ac-curate if the Schur complement is small, but the singularvalue bounds of Theorem 2 are needed to guarantee thatthe approximation retains structural properties of the orig-inal data, such as an accurate approximation of the rankand the spectrum. Furthermore, singular values bounds canbe significantly stronger than the more familiar norm errorbounds that appear in Theorem 1. Theorem 2 provides ageneral framework for singular value bounds, and bound-ing the terms in this theorem provided guidance in the de-sign and development of SRLU. Just as in the case of de-terministic LU with complete pivoting, the sizes of ‖S‖2

σk(LU)

and ‖S‖2σj(LU)

range from moderate to small for almost all

data matrices of practical interest. They, nevertheless, can-not be effectively bounded for a general TRLUCP factor-ization, implying the need for Algorithm 4 to ensure that

An Efficient Algorithm for Low-Rank Approximation

these terms are controlled. While the error bounds in The-orem 3 for the CUR decomposition do not improve uponthe result in Theorem 1, CUR bounds for SRLU specifi-cally will be considerably stronger. Next, we present ourmain theoretical contributions.

4.2. Analysis of the Spectrum-Revealing LUDecomposition

Theorem 4 (SRLU Error Bounds.) For j ≤ k and γ =O (fk

√mn), SRP produces a rank-k SRLU factorization

with

‖Π1AΠT2 − LU‖2 ≤ γσk+1 (A) ,

‖Π1AΠT2 −

(LU)j‖2 ≤ σj+1 (A)

(1 + 2γ σk+1(A)

σj+1(A)

)Theorem 4 is a special case of Theorem 1 for SRLU fac-torizations. For a data matrix with a rapidly decaying spec-trum, the right-hand side of the second inequality is closeto σj+1 (A), a substantial improvement over the sharpnessof the bounds in (Drineas et al., 2008).

Theorem 5 (SRLU Spectral Bound). For 1 ≤ j ≤ k, SRPproduces a rank-k SRLU factorization with

σj (A)

1 + τ σk+1(A)σj(A)

≤ σj(LU)≤ σj (A)

(1 + τ

σk+1 (A)

σj (A)

)for τ ≤ O

(mnk2f3

).

While the worst case upper bound on τ is large, it isdimension-dependent, and j and k may be chosen so thatσk+1(A)σj(A) is arbitrarily small compared to τ . In particular, ifk is the numeric rank of A, then the singular values of theapproximation are numerically equal to those of the data.

These bounds are problem-specific bounds because theirquality depends on the spectrum of the original data, ratherthan universal constants that appear in previous results. Thebenefit of these problem-specific bounds is that an approx-imation of data with a rapidly decaying spectrum is guar-anteed to be high-quality. Furthermore, if σk+1 (A) is notsmall compared to σj (A), then no high-quality low-rankapproximation is possible in the 2 and Frobenius norms.Thus, in this sense, the bounds presented in Theorems 4and 5 are optimal.

Given a high-quality rank-k truncated LU factorization,Theorem 5 ensures that a low-rank approximation of rank` with ` < k of the compressed data is an accurate rank-`approximation of the full data. The proof of this theoremcenters on bounding the terms in Theorems 1 and 2. Exper-iments will show that τ is small in almost all cases.

Stronger results are achieved with the CUR version ofSRLU:

Theorem 6

‖Π1AΠT2 − LMU‖2 ≤ 2γσk+1 (A)

and

‖Π1AΠT2 − LMU‖F ≤ ωσk+1 (A) ,

where γ = O (fk√mn) is the same as in Theorem 4, and

ω = O (fkmn).

Theorem 7 If σ2j (A) > 2‖S‖22 then

σj (A) ≥ σj(LMU

)≥ σj (A)

√1− 2γ

(σk+1 (A)

σj (A)

)2

for γ = O(mnk2f2

)and f is an input parameter control-

ling a tradeoff of quality vs. speed as before.

As before, the constants are small in practice. Observe thatfor most real data matrices, their singular values decay withincreasing j. For such matrices this result is significantlystronger than Theorem 5.

5. Experiments5.1. Speed and Accuracy Tests

In Figure 1, the accuracy of our method is compared tothe accuracy of the truncated SVD. Note that SRLU didnot perform any swaps in these experiments. “CUR” isthe CUR version of the output of SRLU. Note that bothmethods exhibits a convergence rate similar to that of thetruncated SVD (TSVD), and so only a constant amount ofextra work is needed to achieve the same accuracy. Whenthe singular values decay slowly, the CUR decompositionprovides a greater accuracy boost. In Figure 2, the run-time of SRLU is compared to that of the truncated SVD, aswell as Subspace Iteration (Gu, 2015). Note that for Sub-space Iteration, we choose iteration parameter q = 0 anddo not measure the time of applying the random projec-tion, in acknowledgement that fast methods exist to apply arandom projection to a data matrix. Also, the block size im-plemented in SRLU is significantly smaller than the blocksize used by the standard software LAPACK, as the size ofthe block size affects the size of the projection. See supple-ment for additional details. All numeric experiments wererun on NERSC’s Edison. For timing experiments, the trun-cated SVD is calculated with PROPACK.

Even more impressive, the factorization stage of SRLU be-comes arbitrarily faster than the standard implementationof the LU decomposition. Although the standard LU de-composition is not a low-rank approximation algorithm, itis known to be roughly 10 times faster than the SVD (Dem-mel, 1997). See appendix for details.

An Efficient Algorithm for Low-Rank Approximation

(a) Spectral Decay = 0.8 (b) Spectral Decay = 0.95

Figure 1. Accuracy experiment on random 1000x1000 matrices with different rates of spectral decay.

(a) Rank-100 Factorizations (b) Time vs. Truncation Rank

Figure 2. Time experiment on various random matrices, and a time experiment on a 1000x1000 matrix with varying truncation ranks.

Table 1. Errors of low-rank approximations of the given targetrank. SRLU is measured using the CUR version of the factor-ization.

DATA k Gaus. SRFT Dual BCH SRLU

S80PIn1 63 3.85 3.80 3.81 2.84deter3 127 9.27 9.30 9.26 8.30lc3d 63 18.39 16.36 15.49 16.94lc3d 78 15.11

Next, we compare SRLU against competing algorithms. In(Ubaru et al., 2015), error-correcting codes are introducedto yield improved accuracy over existing random projec-tion low-rank approximation algorithms. Their algorithm,denoted Dual BCH, is compared against SRLU as well astwo other random projection methods: Gaus., which usesa Gaussian random projection, and SRFT, which uses aFourier transform to apply a random projection. We test thespectral norm error of these algorithms on matrices fromthe sparse matrix collection in (Davis & Hu, 2011).

In Table 1, results for SRLU are averaged over 5 exper-iments. Using tuning parameter f = 5, no swaps wereneeded in all cases. The matrices being tested are sparsematrices from various engineering problems. S80PIn1

is 4,028 by 4,028, deter3 is 7,647 by 21,777, andlp ceria3d (abbreviated lc3d) is 3,576 by 4,400. Note

that SRLU, a more efficient algorithm, provides a better ap-proximation in two of the three experiments. With a littleextra oversampling, a practical assumption due to the speedadvantage, SRLU achieves a competitive quality approx-imation. The oversampling highlights an additional andunique advantage of SRLU over competing algorithms: ifmore accuracy is desired, then the factorization can simplycontinue as needed.

5.2. Sparsity Preservation Tests

The SRLU factorization is tested on sparse, unsymmetricmatrices from (Davis & Hu, 2011). Figure 3 shows thesparsity patterns of the factors of an SRLU factorizationof a sparse data matrix representing a circuit simulation(oscil dcop), as well as a full LU decomposition of thedata. Note that the LU decomposition preserves the spar-sity of the data initially, but the full LU decomposition be-comes dense. Several more experiments are shown in thesupplement.

5.3. Towards Feature Selection

An image processing example is now presented to illustratethe benefit of highlighting important rows and columns se-lection. In Figure 4 an image is compressed to a rank-50 approximation using SRLU. Note that the rows andcolumns chosen overlap with the astronaut and the planet,implying that minimal storage is needed to capture the

An Efficient Algorithm for Low-Rank Approximation

(a) L and U patterns of a low-rank factorization

(b) L and U patterns of the full factorization

Figure 3. The sparsity patterns of the L and U matrices of a rank43 SRLU factorization, followed by the sparsity pattern of the Land U matrices of a full LU decomposition of the same data. Forthe SRLU factorization, the green entries compose the low-rankapproximation of the data. The red entries are the additional dataneeded for an exact factorization.

black background, which composes approximately twothirds of the image. While this result cannot be called fea-ture selection per se, the rows and columns selected high-light where to look for features: rows and/or columns areselected in a higher density around the astronaut, the cur-vature of the planet, and the storm front on the planet.

(a) Original (b) SRLU (c) Rows and Cols.

Figure 4. Image processing example. The original image(NASA), a rank-50 approximation with SRLU, and a highlightof the rows and columns selected by SRLU.

5.4. Online Data Processing

Online SRLU is tested here on the Enron email corpus(Lichman, 2013). The documents were initially reverse-sorted by the usage of the most common word, and thenreverse-sorted by the second most, and this process wasrepeated for the five most common words (the top fivewords were used significantly more than any other), so thatthe most common words occurred most at the end of thecorpus. The data contains 39,861 documents and 28,102

words/terms, and an initial SRLU factorization of rank 20was performed on the first 30K documents. The initial fac-torization contained none of the top five words, but, af-ter adding the remaining documents and updating, the topthree were included in the approximation. The fourth andfifth words ‘market’ and ‘california’ have high covariancewith at least two of the three top words, and so their inclu-sion may be redundant in a low-rank approximation.

6. ConclusionWe have presented SRLU, a low-rank approximationmethod with many desirable properties: efficiency, accu-racy, sparsity-preservation, the ability to be updated, andthe ability to highlight important data features and vari-ables. Extensive theory and numeric experiments have il-lustrated the efficiency and effectiveness of this method.

AcknowledgementsThis research was supported in part by NSF Award CCF-1319312.

ReferencesAizenbud, Y., Shabat, G., and Averbuch, A. Random-

ized lu decomposition using sparse projections. CoRR,abs/1601.04280, 2016.

Batson, J., Spielman, D. A., and Srivastava, N. Twice-ramanujan sparsifiers. SIAM Journal on Computing, 41(6):1704–1721, 2012.

Chan, T. F. Rank revealing qr factorizations. Linear alge-bra and its applications, (88/89):67–82, 1987.

Cheng, H., Gimbutas, Z., Martinsson, P.-G., and Rokhlin,V. On the compression of low rank matrices. SIAM J.Scientific Computing, 26(4):1389–1404, 2005.

Chow, E. and Patel, A. Fine-grained parallel incompletelu factorization. SIAM J. Scientific Computing, 37(2),2015.

Clarkson, K. L. and Woodruff, D. P. Low rank approx-imation and regression in input sparsity time. CoRR,abs/1207.6365, 2012.

Davis, T. A. and Hu, Y. The university of florida sparse ma-trix collection. ACM Transactions on Mathematical Soft-ware, 38:1:1–1:25, 2011. URL http://www.cise.ufl.edu/research/sparse/matrices.

Demmel, J. Applied Numerical Linear Algebra. SIAM,1997.

An Efficient Algorithm for Low-Rank Approximation

Deshpande, A. and Vempala, S. Adaptive sampling and fastlow-rank matrix approximation. In APPROX-RANDOM,volume 4110 of Lecture Notes in Computer Science, pp.292–303. Springer, 2006.

Deshpande, A., Rademacher, L., Vempala, S., and Wang,G. Matrix approximation and projective clustering viavolume sampling. Theory of Computing, 2(12):225–247,2006.

Drineas, P., Mahoney, M. W., and Muthukrishnan, S.Relative-error cur matrix decompositions. SIAM J. Ma-trix Analysis Applications, 30(2):844–881, 2008.

Eckart, C. and Young, G. The approximation of one matrixby another of lower rank. Psychometrika, 1(3):211–218,1936.

Frieze, A. M., Kannan, R., and Vempala, S. Fast monte-carlo algorithms for finding low-rank approximations. J.ACM, 51(6):1025–1041, 2004.

Ghavamzadeh, M., Lazaric, A., Maillard, O.-A., andMunos, R. Lstd with random projections. In NIPS, pp.721–729, 2010.

Gondzio, J. Stable algorithm for updating dense lu factor-ization after row or column exchange and row and col-umn addition or deletion. Optimization: A journal ofMathematical Programming and Operations Research,pp. 7–26, 2007.

Grigori, L., Demmel, J., and Li, X. S. Parallel symbolicfactorization for sparse lu with static pivoting. SIAM J.Scientific Computing, 29(3):1289–1314, 2007.

Gu, M. Subspace iteration randomization and singularvalue problems. SIAM J. Scientific Computing, 37(3),2015.

Gu, M. and Eisenstat, S. C. Efficient algorithms for com-puting a strong rank-revealing qr factorization. SIAM J.Sci. Comput., 17(4):848–869, 1996.

Halko, N., Martinsson, P.-G., and Tropp, J. A. Findingstructure with randomness: Probabilistic algorithms forconstructing approximate matrix decompositions. SIAMReview, 53(2):217–288, 2011.

Higham, N. J. and Relton, S. D. Estimating the largestentries of a matrix. 2015. URL http://eprints.ma.man.ac.uk/.

Jaderberg, M., Vedaldi, A., and Zisserman, A. Speeding upconvolutional neural networks with low rank expansions.CoRR, abs/1405.3866, 2014.

Johnson, W. B. and Lindenstrauss, J. Extensions of lip-schitz mappings into a hilbert space. ContemporaryMathematics, 26:189–206, 1984.

Khabou, A., Demmel, J., Grigori, L., and Gu, M. Lu factor-ization with panel rank revealing pivoting and its com-munication avoiding version. SIAM J. Matrix AnalysisApplications, 34(3):1401–1429, 2013.

Kirkpatrick, J., Pascanu, R., Rabinowitz, N. C., Veness,J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J.,Ramalho, T., Grabska-Barwinska, A., Hassabis, D.,Clopath, C., Kumaran, D., and Hadsell, R. Overcom-ing catastrophic forgetting in neural networks. Pro-ceedings of the National Academy of Sciences, 114(13):3521–3526, 2017.

Krummenacher, G., McWilliams, B., Kilcher, Y., Buh-mann, J. M., and Meinshausen, N. Scalable adaptivestochastic optimization using random projections. InNIPS, pp. 1750–1758, 2016.

Liberty, E., Woolfe, F., Martinsson, P.G., Rokhlin, V., andTygert, M. Randomized algorithms for the low-rank ap-proximation of matrices. Proceedings of the NationalAcademy of Sciences, 104(51):20167, 2007.

Lichman, M. UCI machine learning repository, 2013. URLhttp://archive.ics.uci.edu/ml.

Luo, H., Agarwal, A., Cesa-Bianchi, N., and Langford, J.Efficient second order online learning by sketching. InNIPS, pp. 902–910, 2016.

Mahoney, M. W. and Drineas, P. Cur matrix decompo-sitions for improved data analysis. Proceedings of theNational Academy of Sciences, 106(3):697–702, 2009.

Martinsson, P.-G., Rokhlin, V., and Tygert, M. A random-ized algorithm for the approximation of matrices. Tech.Rep., Yale University, Department of Computer Science,(1361), 2006.

Melgaard, C. and Gu, M. Gaussian elimination with ran-domized complete pivoting. CoRR, abs/1511.08528,2015.

Miranian, L. and Gu, M. Stong rank revealing lu factor-izations. Linear Algebra and its Applications, 367:1–16,2003.

NASA. Nasa celebrates 50 years of spacewalk-ing. URL https://www.nasa.gov/image-feature/nasa-celebrates-50-years-of-spacewalking. Accessed on August 22, 2015.Published on June 3, 2015. Original photograph fromFebruary 7, 1984.

An Efficient Algorithm for Low-Rank Approximation

Papadimitriou, C. H., Raghavan, P., Tamaki, H., and Vem-pala, S. Latent semantic indexing: A probabilistic anal-ysis. J. Comput. Syst. Sci., 61(2):217–235, 2000.

Sarlos, T. Improved approximation algorithms for largematrices via random projections. In FOCS, pp. 143–152.IEEE Computer Society, 2006.

Stewart, G. W. An updating algorithm for subspace track-ing. IEEE Trans. Signal Processing, 40(6):1535–1541,1992.

Ubaru, S., Mazumdar, A., and Saad, Y. Low rank approxi-mation using error correcting coding matrices. In ICML,volume 37, pp. 702–710, 2015.

Wang, W. Y., Mehdad, Y., Radev, D. R., and Stent, A. Alow-rank approximation approach to learning joint em-beddings of news stories and images for timeline sum-marization. pp. 58–68, 2016.

Woolfe, F., Liberty, E., Rokhlin, V., and Tygert, M. A fastrandomized algorithm for the approximation of matrices.Applied and Computational Harmonic Analysis, 25(3):335–366, 2008.