randomized svd, curdecomposition, and spsd matrix...
TRANSCRIPT
![Page 1: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/1.jpg)
Randomized SVD, CUR Decomposition,and SPSD Matrix Approximation
Shusen Wang
![Page 2: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/2.jpg)
Outline
• CX Decomposition & Approximate SVD• CUR Decomposition• SPSD Matrix Approximation
![Page 3: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/3.jpg)
• Givenanymatrix𝐀 ∈ ℝ$×&
• TheCXdecompositionof𝐀1. Sketching:𝐂 = 𝐀𝐏 ∈ ℝ$×*2. Find𝐗 suchthat𝐀 ≈ 𝐂𝐗
• E.g.𝐗⋆ = argmin𝐗 𝐀 − 𝐂𝐗 56 = 𝐂7𝐀
• Itcosts𝑂 𝑚𝑛𝑐
• CXdecomposition⇔approximateSVD𝐀 ≈ 𝐂𝐗 = 𝐔G𝚺G𝐕GJ𝐗 = 𝐔G𝐙 = 𝐔G𝐔L𝚺L𝐕LJ
CX Decomposition
![Page 4: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/4.jpg)
• Let the sketching matrix 𝐏 ∈ ℝ&×* be defined in the table.
• minWXYZ 𝐗 [\
𝐀 − 𝐂𝐗 ]6 ≤ 1 + 𝜖 𝐀 − 𝐀\ ]
6
Uniformsampling Leveragescoresampling
Gaussianprojection
SRHT Countsketch
c ≥ O 𝜈𝑘 log 𝑘 +1𝜖 O 𝑘 log 𝑘 +
1𝜖
O𝑘𝜖 O 𝑘 + log 𝑛 log 𝑘 +
1𝜖
O 𝑘6 +𝑘𝜖
𝜈 isthecolumncoherenceof𝐀Z
CX Decomposition
![Page 5: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/5.jpg)
CX Decomposition ⇔ Approximate SVD
• CXdecomposition⇔approximateSVD
𝐀 ≈ 𝐂𝐗 = 𝐔G𝚺G𝐕GJ𝐗 = 𝐔G𝐙 = 𝐔G𝐔L𝚺L𝐕LJ
![Page 6: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/6.jpg)
CX Decomposition ⇔ Approximate SVD
• CXdecomposition⇔approximateSVD
𝐀 ≈ 𝐂𝐗 = 𝐔G𝚺G𝐕GJ𝐗 = 𝐔G𝐙 = 𝐔G𝐔L𝚺L𝐕LJ
SVD:𝐂 = 𝐔G 𝚺G𝐕GJ ∈ ℝ$×*
Timecost:𝑂(𝑚𝑐6)
![Page 7: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/7.jpg)
CX Decomposition ⇔ Approximate SVD
• CXdecomposition⇔approximateSVD
𝐀 ≈ 𝐂𝐗 = 𝐔G𝚺G𝐕GJ𝐗 = 𝐔G𝐙 = 𝐔G𝐔L𝚺L𝐕LJ
SVD:𝐂 = 𝐔G 𝚺G𝐕GJ ∈ ℝ$×*
Let𝚺G𝐕GJ𝐗 = 𝐙 ∈ ℝ*×&
Timecost:𝑂(𝑚𝑐6 + 𝑛𝑐6)
![Page 8: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/8.jpg)
CX Decomposition ⇔ Approximate SVD
• CXdecomposition⇔approximateSVD
𝐀 ≈ 𝐂𝐗 = 𝐔G𝚺G𝐕GJ𝐗 = 𝐔G𝐙 = 𝐔G𝐔L𝚺L𝐕LJ
SVD:𝐂 = 𝐔G 𝚺G𝐕GJ ∈ ℝ$×*
Let𝚺G𝐕GJ𝐗 = 𝐙 ∈ ℝ*×&
SVD:𝐙 = 𝐔L𝚺L𝐕LJ ∈ ℝ*×&
Timecost:𝑂(𝑚𝑐6 + 𝑛𝑐6 + 𝑛𝑐6)
![Page 9: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/9.jpg)
CX Decomposition ⇔ Approximate SVD
• CXdecomposition⇔approximateSVD
𝐀 ≈ 𝐂𝐗 = 𝐔G𝚺G𝐕GJ𝐗 = 𝐔G𝐙 = 𝐔G𝐔L𝚺L𝐕LJ
SVD:𝐂 = 𝐔G 𝚺G𝐕GJ ∈ ℝ$×*
Let𝚺G𝐕GJ𝐗 = 𝐙 ∈ ℝ*×&
SVD:𝐙 = 𝐔L𝚺L𝐕LJ ∈ ℝ*×&
𝑚×𝑠 matrixwithorthonormalcolumns
diagonalmatrix
𝑠×𝑛 matrixwithorthonormalrows
Timecost:𝑂(𝑚𝑐6 + 𝑛𝑐6 + 𝑛𝑐6 + 𝑚𝑐6)
![Page 10: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/10.jpg)
CX Decomposition ⇔ Approximate SVD
• CXdecomposition⇔approximateSVD
• Done!Approximaterank𝑐 SVD:𝐀 ≈ (𝐔G𝐔L)𝚺L𝐕LJ
𝐀 ≈ 𝐂𝐗 = 𝐔G𝚺G𝐕GJ𝐗 = 𝐔G𝐙 = 𝐔G𝐔L𝚺L𝐕LJ
𝑚×𝑠 matrixwithorthonormalcolumns
diagonalmatrix
𝑠×𝑛 matrixwithorthonormalrows
Timecost:𝑂 𝑚𝑐6 + 𝑛𝑐6 + 𝑛𝑐6 + 𝑚𝑐6 = 𝑂(𝑚𝑐6 + 𝑛𝑐6)
![Page 11: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/11.jpg)
CX Decomposition ⇔ Approximate SVD
• CXdecomposition⇔approximateSVD
• Given𝐀 ∈ ℝ$×& and𝐂 ∈ ℝ$×*,theapproximateSVDcosts• 𝑂 𝑚𝑛𝑐 time• 𝑂 𝑚𝑐 + 𝑛𝑐 memory
![Page 12: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/12.jpg)
CX Decomposition
• TheCXdecompositionof𝐀 ∈ ℝ$×&
• Optimalsolution:𝐗⋆ = argmin𝐗 𝐀 − 𝐂𝐗 56 = 𝐂7𝐀
• Howtomakeitmoreefficient?
![Page 13: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/13.jpg)
CX Decomposition
• TheCXdecompositionof𝐀 ∈ ℝ$×&
• Optimalsolution:𝐗⋆ = argmin𝐗 𝐀 − 𝐂𝐗 56 = 𝐂7𝐀
• Howtomakeitmoreefficient?Aregressionproblem!
![Page 14: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/14.jpg)
Fast CX Decomposition
• FastCX [Drineas, Mahoney, Muthukrishnan, 2008][Clarkson & Woodruff, 2013]
• Drawanothersketchingmatrix𝐒 ∈ ℝ$×m
• Compute𝐗n = argmin𝐗 𝐒o 𝐀 − 𝐂𝐗 56 = 𝐒J𝐂 7 𝐒J𝐀
• Timecost:𝑂 𝑛𝑐𝑠 + TimeOfSketch• When𝑠 = 𝑂q 𝑐/𝜖 ,
𝐀 − 𝐂𝐗n5
6≤ 1 + 𝜖 ⋅ min𝐗 𝐀 − 𝐂𝐗 5
6
![Page 15: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/15.jpg)
Outline
• CX Decomposition & Approximate SVD• CUR Decomposition• SPSD Matrix Approximation
![Page 16: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/16.jpg)
CUR Decomposition
• Sketching• 𝐂 = 𝐀𝐏𝐂 ∈ ℝ$×*
• 𝐑 = 𝐏𝐑J𝐀 ∈ ℝv×&
• Find 𝐔 such that 𝐂𝐔𝐑 ≈ 𝐀• CUR⇔ ApproximateSVD• Inthesamewayas“CX⇔ ApproximateSVD”
![Page 17: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/17.jpg)
CUR Decomposition
• Sketching• 𝐂 = 𝐀𝐏𝐂 ∈ ℝ$×*
• 𝐑 = 𝐏𝐑J𝐀 ∈ ℝv×&
• Find 𝐔 such that 𝐂𝐔𝐑 ≈ 𝐀• CUR⇔ ApproximateSVD• Inthesamewayas“CX⇔ ApproximateSVD”
• 3 types of 𝐔
![Page 18: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/18.jpg)
CUR Decomposition
• Type 1 [Drineas, Mahoney, Muthukrishnan, 2008]:𝐔 = 𝐏𝐑o𝐀𝐏𝐂
7
𝐂𝐀 𝐔 𝐑
![Page 19: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/19.jpg)
CUR Decomposition
• Type 1 [Drineas, Mahoney, Muthukrishnan, 2008]:𝐔 = 𝐏𝐑o𝐀𝐏𝐂
7
• Recall the fast CX decomposition𝐀 ≈ 𝐂𝐗n = 𝐂 𝐏𝐑o𝐂
7 𝐏𝐑o𝐀 = 𝐂𝐔𝐑
![Page 20: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/20.jpg)
CUR Decomposition
• Type 1 [Drineas, Mahoney, Muthukrishnan, 2008]:𝐔 = 𝐏𝐑o𝐀𝐏𝐂
7
• Recall the fast CX decomposition𝐀 ≈ 𝐂𝐗n = 𝐂 𝐏𝐑o𝐂
7 𝐏𝐑o𝐀 = 𝐂𝐔𝐑
![Page 21: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/21.jpg)
CUR Decomposition
• Type 1 [Drineas, Mahoney, Muthukrishnan, 2008]:𝐔 = 𝐏𝐑o𝐀𝐏𝐂
7
• Recall the fast CX decomposition𝐀 ≈ 𝐂𝐗n = 𝐂 𝐏𝐑o𝐂
7 𝐏𝐑o𝐀 = 𝐂𝐔𝐑
• They’re equivalent: 𝐂𝐗n = 𝐂𝐔𝐑
![Page 22: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/22.jpg)
CUR Decomposition
• Type 1 [Drineas, Mahoney, Muthukrishnan, 2008]:𝐔 = 𝐏𝐑o𝐀𝐏𝐂
7
• Recall the fast CX decomposition𝐀 ≈ 𝐂𝐗n = 𝐂 𝐏𝐑o𝐂
7 𝐏𝐑o𝐀 = 𝐂𝐔𝐑
• They’re equivalent: 𝐂𝐗n = 𝐂𝐔𝐑
• Require 𝑐 = 𝑂q \w
and 𝑟 = 𝑂q *w
such that𝐀 − 𝐂𝐔𝐑 5
6 ≤ 1 + 𝜖 𝐀 − 𝐀\ 56
![Page 23: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/23.jpg)
CUR Decomposition
• Type 1 [Drineas, Mahoney, Muthukrishnan, 2008]:𝐔 = 𝐏𝐑𝐓𝐀𝐏𝐂
7
• Efficient• O 𝑟𝑐6 + TimeOfSketch
• Loose bound• Sketch size ∝ 𝜖{6
• Bad empirical performance
![Page 24: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/24.jpg)
CUR Decomposition
• Type 2:OptimalCUR𝐔⋆ = min
𝐔𝐀 − 𝐂𝐔𝐑 ]
6 = 𝐂7𝐀𝐑7
![Page 25: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/25.jpg)
CUR Decomposition
• Type 2:OptimalCUR𝐔⋆ = min
𝐔𝐀 − 𝐂𝐔𝐑 ]
6 = 𝐂7𝐀𝐑7
• Theory [W&Zhang,2013], [Boutsidis & Woodruff, 2014]:• 𝐂 and 𝐑 are selected by the adaptive sampling algorithm• 𝑐 = 𝑂 \
w and 𝑟 = 𝑂 \w
• 𝐀 − 𝐂𝐔𝐑 ]6 ≤ 1 + 𝜖 𝐀 − 𝐀\ 5
6
![Page 26: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/26.jpg)
CUR Decomposition
• Type 2:OptimalCUR𝐔⋆ = min
𝐔𝐀 − 𝐂𝐔𝐑 ]
6 = 𝐂7𝐀𝐑7
• Inefficient• O 𝑚𝑛𝑐 + TimeOfSketch
![Page 27: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/27.jpg)
CUR Decomposition
• Type 3:FastCUR[W, Zhang, Zhang, 2015]
• Draw 2 sketching matrices 𝐒𝐂 and 𝐒𝐑• Solve the problem
𝐔n = min𝐔
𝑺𝑪o 𝐀 − 𝐂𝐔𝐑 𝐒𝐑 ]
6= 𝐒𝐂J𝐂
7 𝐒𝐂o𝐀𝐒𝑹 𝐑𝐒𝐑 7
• Intuition?
![Page 28: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/28.jpg)
CUR Decomposition
• The optimal𝐔matrixis obtained by the optimization problem𝐔⋆ = min
𝐔𝐂𝐔𝐑 − 𝐀 ]
6
![Page 29: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/29.jpg)
CUR Decomposition
• Approximately solve the optimization problem, e.g. by columnselection
![Page 30: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/30.jpg)
CUR Decomposition
• Solve the small scale problem
![Page 31: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/31.jpg)
CUR Decomposition
• Type 3:FastCUR[W, Zhang, Zhang, 2015]
• Draw 2 sketching matrices 𝐒𝐂 ∈ ℝ$×m� and 𝐒𝐑 ∈ ℝ&×m�• Solve the problem
𝐔n = min𝐔
𝐒𝑪o 𝐀 − 𝐂𝐔𝐑 𝐒𝐑 ]
6= 𝐒𝐂J𝐂
7 𝐒𝐂o𝐀𝐒𝑹 𝐑𝐒𝐑 7
• Theory• 𝑠* = 𝑂 *
w and sv = 𝑂 vw
• 𝐀 − 𝐂𝐔n𝐑]
6≤ 1 + 𝜖 ⋅ min
𝐔𝐀 − 𝐂𝐔𝐑 ]
6
![Page 32: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/32.jpg)
CUR Decomposition
• Type 3:FastCUR[W, Zhang, Zhang, 2015]
• Draw 2 sketching matrices 𝐒𝐂 ∈ ℝ$×m� and 𝐒𝐑 ∈ ℝ&×m�• Solve the problem
𝐔n = min𝐔
𝐒𝑪o 𝐀 − 𝐂𝐔𝐑 𝐒𝐑 ]
6= 𝐒𝐂J𝐂
7 𝐒𝐂o𝐀𝐒𝑹 𝐑𝐒𝐑 7
• Efficient• 𝑂 𝑠*𝑠v 𝑐 + 𝑟 + TimeOfSketch
• Good empirical performance
![Page 33: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/33.jpg)
Type 2:OptimalCUROriginal
Type 1:FastCX Type 3:FastCUR𝑠* = 2𝑐, 𝑠v = 2𝑟
Type 3:FastCUR𝑠* = 4𝑐, 𝑠v = 4𝑟
𝐀:𝑚 = 1920𝑛 = 1168
𝐂 and𝐑:• 𝑐 = 𝑟 = 100• uniformsampling
![Page 34: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/34.jpg)
Conclusions
• ApproximatetruncatedSVD• CXdecomposition• CURdecomposition(3types)
• FastCURisthebest
![Page 35: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/35.jpg)
Outline
• CX Decomposition & Approximate SVD• CUR Decomposition• SPSD Matrix Approximation
![Page 36: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/36.jpg)
Motivation 1: Kernel Matrix
• Given𝑛 samples𝐱�,⋯ , 𝐱& ∈ ℝ� andkernelfunction𝜅 ⋅,⋅ .• E.g.GaussianRBFkernel
𝜅 𝐱�, 𝐱� = exp −𝐱� − 𝐱� 6
6
𝜎6 .
• Computingthekernelmatrix𝐊 ∈ ℝ&×&• where𝑘�� = 𝜅 𝐱�, 𝐱�• costsO(𝑛6𝑑) time
![Page 37: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/37.jpg)
Motivation 1: Kernel Matrix
• Given𝑛 samples𝐱�,⋯ , 𝐱& ∈ ℝ� andkernelfunction𝜅 ⋅,⋅ .• E.g.GaussianRBFkernel
𝜅 𝐱�, 𝐱� = exp −𝐱� − 𝐱� 6
6
𝜎6 .
• Computingthekernelmatrix𝐊 ∈ ℝ&×&• where𝑘�� = 𝜅 𝐱�, 𝐱�• costs𝑂(𝑛6𝑑) time
![Page 38: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/38.jpg)
Motivation 2: Matrix Inversion
• Solvethelinearsystem𝐊 + 𝛼𝐈& 𝐰 = 𝐲
tofind𝐰 ∈ ℝ&.• ItcostsO(𝑛�) timeandO(𝑛6)memory.• Performedby• Gaussianprocessregression(equivalently,kernelridgeregression)• LeastsquareskernelSVM
• 𝐊 ∈ ℝ&×& isthekernelmatrix• 𝐲 = 𝑦�,⋯ , 𝑦& ∈ ℝ& containsthelabels
![Page 39: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/39.jpg)
Motivation 2: Matrix Inversion
• Solvethelinearsystem𝐊 + 𝛼𝐈& 𝐰 = 𝐲
tofind𝐰 ∈ ℝ&.• Solution:𝐰⋆ = 𝐊 + 𝛼𝐈& {�𝐲
![Page 40: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/40.jpg)
Motivation 2: Matrix Inversion
• Solvethelinearsystem𝐊 + 𝛼𝐈& 𝐰 = 𝐲
tofind𝐰 ∈ ℝ&.• Solution:𝐰⋆ = 𝐊 + 𝛼𝐈& {�𝐲• Itcosts• 𝑂(𝑛�) time• 𝑂(𝑛6)memory.
![Page 41: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/41.jpg)
Motivation 2: Matrix Inversion
• Solvethelinearsystem𝐊 + 𝛼𝐈& 𝐰 = 𝐲
tofind𝐰 ∈ ℝ&.• Solution:𝐰⋆ = 𝐊 + 𝛼𝐈& {�𝐲• Itcosts• 𝑂(𝑛�) time• 𝑂(𝑛6)memory.
• Performedby• Kernelridgeregression• LeastsquareskernelSVM
![Page 42: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/42.jpg)
Motivation 3: Eigenvalue Decomposition
• Findthetop𝑘 (≪ 𝑛)eigenvectorsof𝐊.• Itcosts• 𝑂q(𝑛6𝑘) time• 𝑂(𝑛6)memory.
![Page 43: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/43.jpg)
Motivation 3: Eigenvalue Decomposition
• Findthetop𝑘 (≪ 𝑛)eigenvectorsof𝐊.• Itcosts• 𝑂q(𝑛6𝑘) time• 𝑂(𝑛6)memory.
• Performedby• KernelPCA(𝑘 isthetargetrank)• Manifoldlearning (𝑘 isthetargetrank)
![Page 44: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/44.jpg)
Computational Challenges
• Timecosts• Computingkernelmatrix:𝑂(𝑛6𝑑)• Matrixinversion:𝑂(𝑛�)• Rank𝑘 eigenvaluedecomposition:𝑂(𝑛6𝑘)
![Page 45: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/45.jpg)
Computational Challenges
• Timecosts• Computingkernelmatrix:𝑂(𝑛6𝑑)• Matrixinversion:𝑂(𝑛�)• Rank𝑘 eigenvaluedecomposition:𝑂(𝑛6𝑘)
Atleastquadratictime!
![Page 46: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/46.jpg)
Computational Challenges
• Timecosts• Computingkernelmatrix:𝑂(𝑛6𝑑)• Matrixinversion:𝑂(𝑛�)• Rank𝑘 eigenvaluedecomposition:𝑂(𝑛6𝑘)
• Memorycosts• Inversionandeigenvaluedecomposition:𝑂(𝑛6)
![Page 47: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/47.jpg)
Computational Challenges
• Timecosts• Computingkernelmatrix:𝑂(𝑛6𝑑)• Matrixinversion:𝑂(𝑛�)• Rank𝑘 eigenvaluedecomposition:𝑂(𝑛6𝑘)
• Memorycosts• Inversionandeigenvaluedecomposition:𝑂(𝑛6)• Because
• thenumericalalgorithmsarepass-inefficient• è form𝐊 andkeepitinmemory
![Page 48: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/48.jpg)
Computational Challenges
• Timecosts• Computingkernelmatrix:𝑂(𝑛6𝑑)• Matrixinversion:𝑂(𝑛�)• Rank𝑘 eigenvaluedecomposition:𝑂(𝑛6𝑘)
• Memorycosts• Inversionandeigenvaluedecomposition:𝑂(𝑛6)• Because
• thenumericalalgorithmsarepass-inefficient• è form𝐊 andkeepitinmemory
When𝑛 = 10�,the𝑛×𝑛 matrixcosts80GBmemory!
![Page 49: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/49.jpg)
How to Speedup?
• Efficientlyformthelow-rankapproximation𝐊 ≈ 𝐂𝐔𝐂J
𝐊 𝐂 𝐂J𝐔
![Page 50: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/50.jpg)
How to Speedup?
• Efficientlyformthelow-rankapproximation𝐊 ≈ 𝐂𝐔𝐂J
• Equivalent𝐊 ≈ 𝐋𝐋J
![Page 51: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/51.jpg)
Efficient Matrix Inversion
• Solvethelinearsystem 𝐊 + 𝛼𝐈& 𝐰 = 𝐲:
• Replace𝐊 by𝐋𝐋J:𝐰⋆ = 𝐊 + 𝛼𝐈& {�𝐲 ≈ 𝐋𝐋J + 𝛼𝐈&{�𝐲
![Page 52: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/52.jpg)
Efficient Matrix Inversion
• Approximatelysolvethelinearsystem 𝐊 + 𝛼𝐈& 𝐰 = 𝐲:
• Replace𝐊 by𝐋𝐋J:𝐰⋆ = 𝐊 + 𝛼𝐈& {�𝐲 ≈ 𝐋𝐋J + 𝛼𝐈&{�𝐲
![Page 53: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/53.jpg)
Efficient Matrix Inversion
• Approximatelysolvethelinearsystem 𝐊 + 𝛼𝐈& 𝐰 = 𝐲
• Replace𝐊 by𝐋𝐋J:𝐰⋆ = 𝐊 + 𝛼𝐈& {�𝐲 ≈ 𝐋𝐋J + 𝛼𝐈&{�𝐲
• Expandtheinversion bytheWoodburyidentity
![Page 54: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/54.jpg)
Efficient Matrix Inversion
• Approximatelysolvethelinearsystem 𝐊 + 𝛼𝐈& 𝐰 = 𝐲
• Replace𝐊 by𝐋𝐋J:𝐰⋆ = 𝐊 + 𝛼𝐈& {�𝐲 ≈ 𝐋𝐋J + 𝛼𝐈&{�𝐲
• Expandtheinversion bytheWoodburyidentity
𝐀 + 𝐁𝐂𝐃 {� = 𝐀{� − 𝐀{�𝐁 𝐂{� + 𝐃𝐀{�𝐁 {�𝐃𝐀{�
![Page 55: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/55.jpg)
Efficient Matrix Inversion
• Approximatelysolvethelinearsystem 𝐊 + 𝛼𝐈& 𝐰 = 𝐲
• Replace𝐊 by𝐋𝐋J:𝐰⋆ = 𝐊 + 𝛼𝐈& {�𝐲 ≈ 𝐋𝐋J + 𝛼𝐈&{�𝐲
• Expandtheinversion bytheWoodburyidentity𝐰⋆ ≈ 𝛼{�𝐲 + 𝛼{�𝐋 𝛼𝐈 + 𝐋o𝐋 {�𝐋𝐓𝐲
![Page 56: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/56.jpg)
Efficient Matrix Inversion
• Approximatelysolvethelinearsystem 𝐊 + 𝛼𝐈& 𝐰 = 𝐲
• Replace𝐊 by𝐋𝐋J:𝐰⋆ = 𝐊 + 𝛼𝐈& {�𝐲 ≈ 𝐋𝐋J + 𝛼𝐈&{�𝐲
• ExpandtheinversionbytheWoodburyidentity𝐰⋆ ≈ 𝛼{�𝐲 + 𝛼{�𝐋 𝛼𝐈 + 𝐋o𝐋 {�𝐋𝐓𝐲
• Timecost:𝑂 𝑛𝑐6
Linearin𝑛,muchbetterthan𝑂 𝑛�
![Page 57: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/57.jpg)
Efficient Eigenvalue Decomposition
• Approximatelycomputethe𝑘-eigenvaluedecompositionof𝐊• SVD:𝐋 = 𝐔𝐋𝚺𝐋𝐕𝐋• 𝐊 ≈ 𝐋𝐋𝐓 = 𝐔𝐋𝚺𝐋𝟐𝐔𝐋𝐓
• Approximate𝑘- eigenvaluedecompositionof𝐊• eigenvectors:thefirst𝑘 vectorsin𝐔𝐋• eigenvalues:thefirst𝑘 vectorsdiagonalentriesin𝚺𝐋
![Page 58: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/58.jpg)
Efficient Eigenvalue Decomposition
• Approximatelycomputethe𝑘-eigenvaluedecompositionof𝐊• SVD:𝐋 = 𝐔𝐋𝚺𝐋𝐕𝐋• 𝐊 ≈ 𝐋𝐋𝐓 = 𝐔𝐋𝚺𝐋𝟐𝐔𝐋𝐓
• Approximate𝑘- eigenvaluedecompositionof𝐊• eigenvectors:thefirst𝑘 vectorsin𝐔𝐋
• Timecost:𝑂 𝑛𝑐6
![Page 59: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/59.jpg)
Efficient Eigenvalue Decomposition
• Approximatelycomputethe𝑘-eigenvaluedecompositionof𝐊• SVD:𝐋 = 𝐔𝐋𝚺𝐋𝐕𝐋• 𝐊 ≈ 𝐋𝐋𝐓 = 𝐔𝐋𝚺𝐋𝟐𝐔𝐋𝐓
• Approximate𝑘- eigenvaluedecomposition of𝐊• eigenvectors:thefirst𝑘 vectorsin𝐔𝐋
• Timecost:𝑂 𝑛𝑐6• Muchlowerthan𝑂q 𝑛6𝑘
![Page 60: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/60.jpg)
Sketching Based Models
• Howtofindsuchanapproximation?
𝐊 ≈ 𝐂𝐔𝐂J
𝐊 𝐂 𝐂J𝐔
![Page 61: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/61.jpg)
Sketching Based Models
• Howtofindsuchanapproximation?
• SketchingbasedMethods:𝐂 = 𝐊𝐒 ∈ ℝ&×* isasketchof𝐊.• 𝐒 ∈ ℝ&×* canbecolumnselectionorrandomprojectionmatrix
𝐊 ≈ 𝐂𝐔𝐂J
![Page 62: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/62.jpg)
Sketching Based Models
• Howtofindsuchanapproximation?
• SketchingbasedMethods:𝐂 = 𝐊𝐒 ∈ ℝ&×* isasketchof𝐊.• 𝐒 ∈ ℝ&×* canbecolumnselectionorrandomprojectionmatrix
• Threemethods:• Theprototypemodel[HMT11,WZ13,WLZ16]
• Thefastmodel[WZZ15]
• TheNyström method [WS15,GM13]
𝐊 ≈ 𝐂𝐔𝐂J
![Page 63: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/63.jpg)
The Prototype Model
• Objective:𝐊 ≈ 𝐂𝐔𝐂J
• Minimizetheapproximationerrorby𝐔⋆ = argmin
𝐔𝐊 − 𝐂𝐔𝐂J
]
6= 𝐂7𝐊 𝐂7 J.
![Page 64: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/64.jpg)
The Prototype Model
• Objective:𝐊 ≈ 𝐂𝐔𝐂J
• Minimizetheapproximationerrorby𝐔⋆ = argmin
𝐔𝐊 − 𝐂𝐔𝐂J
]
6= 𝐂7𝐊 𝐂7 J.
Extension of the random SVD to SPSD matrix [HMT11]
![Page 65: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/65.jpg)
The Prototype Model
• Objective:𝐊 ≈ 𝐂𝐔𝐂J
• Minimizetheapproximationerrorby𝐔⋆ = argmin
𝐔𝐊 − 𝐂𝐔𝐂J
]
6= 𝐂7𝐊 𝐂7 J.
• Time:𝑂(𝑛6𝑐)
• Thetimecomplexityisnearlythesametothe𝑘-eigenvaluedecomposition.• Itismuchfasterthanthe𝑘- eigenvaluedecompositioninpractice.
![Page 66: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/66.jpg)
The Prototype Model
• Objective:𝐊 ≈ 𝐂𝐔𝐂J
• Minimizetheapproximationerrorby𝐔⋆ = argmin
𝐔𝐊 − 𝐂𝐔𝐂J
]
6= 𝐂7𝐊 𝐂7 J.
• Time:𝑂(𝑛6𝑐)• #Passes:one
![Page 67: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/67.jpg)
The Prototype Model
• Objective:𝐊 ≈ 𝐂𝐔𝐂J
• Minimizetheapproximationerrorby𝐔⋆ = argmin
𝐔𝐊 − 𝐂𝐔𝐂J
]
6= 𝐂7𝐊 𝐂7 J.
• Time:𝑂(𝑛6𝑐)• #Passes:one• Memory:𝑂(𝑛𝑐)
• Put𝑘�� inmemoryonlywhenitisvisited• Keep𝐂7 inmemory
![Page 68: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/68.jpg)
The Prototype Model
• ErrorBound• 𝑘 ≪ 𝑛 isarbitraryinteger• 𝐏 samples𝑐 = 𝑂 \
w columns by adaptive sampling
• 𝔼 𝐊 − 𝐂𝐔⋆𝐂J]
6≤ 1 + 𝜖 𝐊 − 𝐊\ ]
6
![Page 69: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/69.jpg)
The Prototype Model
• Limitations• 𝐔⋆ = 𝐂7𝐊 𝐂7 J
• Timecostis𝑂 𝑛6𝑐• Requiresobservingthewholeof𝐊
![Page 70: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/70.jpg)
The Prototype Model
• Prototypemodel:𝐊 ≈ 𝐂𝐔⋆𝐂J,where𝐔⋆ = argmin
𝐔𝐊 − 𝐂𝐔𝐂J
5
6.
𝐊 𝐂 𝐂J𝐔
![Page 71: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/71.jpg)
The Fast Model
• Column/rowselection• Form𝐏J𝐊𝐏 and𝐏J𝐂
𝐏J𝐊𝐏 𝐏J𝐂 𝐂J𝐏𝐔
![Page 72: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/72.jpg)
The Fast Model
• Column/rowselection• Form𝐏J𝐊𝐏 and𝐏J𝐂
𝐏J𝐊𝐏 𝐏J𝐂 𝐂J𝐏𝐔
![Page 73: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/73.jpg)
The Fast Model
• 𝐊 ≈ 𝐂𝐔n 𝐂J, where𝐔n = argmin
𝐔𝐏J 𝐊 − 𝐂𝐔𝐂J 𝐏
𝑭
𝟐.
𝐏J𝐊𝐏 𝐏J𝐂 𝐂J𝐏𝐔
![Page 74: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/74.jpg)
The Fast Model• Prototypemodel: 𝐔⋆ = argmin
𝐔𝐊 − 𝐂𝐔𝐂J
5
6= 𝐂7𝐊 𝐂7 J
• Fastmodel:𝐔n = argmin𝐔
𝐏J 𝐊 − 𝐂𝐔𝐂J 𝐏]
𝟐= 𝐏J𝐂 7(𝐏J𝐊𝐏) 𝐂J𝐏 7.
![Page 75: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/75.jpg)
The Fast Model• Prototypemodel: 𝐔⋆ = argmin
𝐔𝐊 − 𝐂𝐔𝐂J
5
6= 𝐂7𝐊 𝐂7 J
• Fastmodel:𝐔n = argmin𝐔
𝐏J 𝐊 − 𝐂𝐔𝐂J 𝐏]
𝟐= 𝐏J𝐂 7(𝐏J𝐊𝐏) 𝐂J𝐏 7.
• Theory• 𝑝 = 𝑂 &*
w�
• 𝐏 is column selection matrix (according to the row leverage scores of 𝐂)
• Then 𝐊 − 𝐂𝐔n𝐂J]
6≤ 1 + 𝜖 𝐊 − 𝐂𝐔⋆𝐂J
]
6
Thefastermodelisnearlyasgoodastheprototypemodel!
![Page 76: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/76.jpg)
The Fast Model• Prototypemodel: 𝐔⋆ = argmin
𝐔𝐊 − 𝐂𝐔𝐂J
5
6= 𝐂7𝐊 𝐂7 J
• Fastmodel:𝐔n = argmin𝐔
𝐏J 𝐊 − 𝐂𝐔𝐂J 𝐏]
𝟐= 𝐏J𝐂 7(𝐏J𝐊𝐏) 𝐂J𝐏 7.
• Theory• 𝑝 = 𝑂 &*
w�
• 𝐏 is column selection matrix (according to the row leverage scores of 𝐂)
• Then 𝐊 − 𝐂𝐔n𝐂J]
6≤ 1 + 𝜖 𝐊 − 𝐂𝐔⋆𝐂J
]
6
• Overalltimecost:𝑂 𝑝6𝑐 + 𝑛𝑐6 = 𝑂 𝑛𝑐�/𝜖linearin𝑛
![Page 77: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/77.jpg)
The Nyström Method
• 𝐒 (𝑛×𝑐):columnselectionmatrix• 𝐂 = 𝐊𝐒 (𝑛×𝑐),𝐖 = 𝐒J𝐊𝐒 = 𝐒J𝐂 (𝑐×𝑐)• TheNyström method:𝐊 ≈ 𝐂𝐖7𝐂J
𝐊
![Page 78: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/78.jpg)
The Nyström Method
• 𝐒 (𝑛×𝑐):columnselectionmatrix• 𝐂 = 𝐊𝐒 (𝑛×𝑐),𝐖 = 𝐒J𝐊𝐒 = 𝐒J𝐂 (𝑐×𝑐)• TheNyström method:𝐊 ≈ 𝐂𝐖7𝐂J
𝐊 𝐂 𝐂J
![Page 79: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/79.jpg)
The Nyström Method
• 𝐒 (𝑛×𝑐):columnselectionmatrix• 𝐂 = 𝐊𝐒 (𝑛×𝑐),𝐖 = 𝐒J𝐊𝐒 = 𝐒J𝐂 (𝑐×𝑐)• TheNyström method:𝐊 ≈ 𝐂𝐖7𝐂J
𝐊 𝐂 𝐂J𝐖
![Page 80: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/80.jpg)
The Nyström Method
• 𝐒 (𝑛×𝑐):columnselectionmatrix• 𝐂 = 𝐊𝐒 (𝑛×𝑐),𝐖 = 𝐒J𝐊𝐒 = 𝐒J𝐂 (𝑐×𝑐)• TheNyström method:𝐊 ≈ 𝐂𝐖7𝐂J
𝐊 𝐂 𝐂J𝐖7
![Page 81: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/81.jpg)
The Nyström Method
• 𝐒 (𝑛×𝑐):columnselectionmatrix• 𝐂 = 𝐊𝐒 (𝑛×𝑐),𝐖 = 𝐒J𝐊𝐒 = 𝐒J𝐂 (𝑐×𝑐)• TheNyström method:𝐊 ≈ 𝐂𝐖7𝐂J
• Newexplanation:• Recallthefastmodel:𝐗n = argmin
𝐗𝐏J 𝐊 − 𝐂𝐗𝐂J 𝐏
5
6
• Setting𝐏 = 𝐒,then𝐗n = argmin
𝐗𝐒J 𝐊 − 𝐂𝐗𝐂J 𝐒
5
6
= 𝐒J𝐂 7 𝐒J𝐊𝐒 𝐂J𝐒 7
= 𝐖7𝐖𝐖7 = 𝐖7
![Page 82: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/82.jpg)
The Nyström Method
• 𝐒 (𝑛×𝑐):columnselectionmatrix• 𝐂 = 𝐊𝐒 (𝑛×𝑐),𝐖 = 𝐒J𝐊𝐒 = 𝐒J𝐂 (𝑐×𝑐)• TheNyström method:𝐊 ≈ 𝐂𝐖7𝐂J
• Newexplanation:• Recallthefastmodel:𝐗n = argmin
𝐗𝐏J 𝐊 − 𝐂𝐗𝐂J 𝐏
5
6
• Setting𝐏 = 𝐒,then𝐗n = argmin
𝐗𝐒J 𝐊 − 𝐂𝐗𝐂J 𝐒
5
6
= 𝐒J𝐂 7 𝐒J𝐊𝐒 𝐂J𝐒 7
= 𝐖7𝐖𝐖7 = 𝐖7
![Page 83: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/83.jpg)
The Nyström Method
• 𝐒 (𝑛×𝑐):columnselectionmatrix• 𝐂 = 𝐊𝐒 (𝑛×𝑐),𝐖 = 𝐒J𝐊𝐒 = 𝐒J𝐂 (𝑐×𝑐)• TheNyström method:𝐊 ≈ 𝐂𝐖7𝐂J
• Newexplanation:• Recallthefastmodel:𝐗n = argmin
𝐗𝐏J 𝐊 − 𝐂𝐗𝐂J 𝐏
5
6
• Setting𝐏 = 𝐒,then𝐗n = argmin
𝐗𝐒J 𝐊 − 𝐂𝐗𝐂J 𝐒
5
6
= 𝐒J𝐂 7 𝐒J𝐊𝐒 𝐂J𝐒 7
= 𝐖7𝐖𝐖7 = 𝐖7
![Page 84: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/84.jpg)
The Nyström Method
• 𝐒 (𝑛×𝑐):columnselectionmatrix• 𝐂 = 𝐊𝐒 (𝑛×𝑐),𝐖 = 𝐒J𝐊𝐒 = 𝐒J𝐂 (𝑐×𝑐)• TheNyström method:𝐊 ≈ 𝐂𝐖7𝐂J
• Newexplanation:• Recallthefastmodel:𝐗n = argmin
𝐗𝐏J 𝐊 − 𝐂𝐗𝐂J 𝐏
5
6
• Setting𝐏 = 𝐒,then𝐗n = argmin
𝐗𝐒J 𝐊 − 𝐂𝐗𝐂J 𝐒
5
6
= 𝐒J𝐂 7 𝐒J𝐊𝐒 𝐂J𝐒 7
= 𝐖7𝐖𝐖7 = 𝐖7
![Page 85: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/85.jpg)
The Nyström Method
• 𝐒 (𝑛×𝑐):columnselectionmatrix• 𝐂 = 𝐊𝐒 (𝑛×𝑐),𝐖 = 𝐒J𝐊𝐒 = 𝐒J𝐂 (𝑐×𝑐)• TheNyström method:𝐊 ≈ 𝐂𝐖7𝐂J
• Newexplanation:• Recallthefastmodel:𝐗n = argmin
𝐗𝐏J 𝐊 − 𝐂𝐗𝐂J 𝐏
5
6
• Setting𝐏 = 𝐒,then𝐗n = argmin
𝐗𝐒J 𝐊 − 𝐂𝐗𝐂J 𝐒
5
6
= 𝐒J𝐂 7 𝐒J𝐊𝐒 𝐂J𝐒 7
= 𝐖7𝐖𝐖7 = 𝐖7
![Page 86: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/86.jpg)
The Nyström Method
• 𝐒 (𝑛×𝑐):columnselectionmatrix• 𝐂 = 𝐊𝐒 (𝑛×𝑐),𝐖 = 𝐒J𝐊𝐒 = 𝐒J𝐂 (𝑐×𝑐)• TheNyström method:𝐊 ≈ 𝐂𝐖7𝐂J
• Newexplanation:• Recallthefastmodel:𝐗n = argmin
𝐗𝐏J 𝐊 − 𝐂𝐗𝐂J 𝐏
5
6
• Setting𝐏 = 𝐒,then𝐗n = argmin
𝐗𝐒J 𝐊 − 𝐂𝐗𝐂J 𝐒
5
6
= 𝐒J𝐂 7 𝐒J𝐊𝐒 𝐂J𝐒 7
= 𝐖7𝐖𝐖7 = 𝐖7
• TheNystrommethodisspecialinstanceofthefastmodel.
• Itisapproximatesolutiontotheprototypemodel
![Page 87: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/87.jpg)
The Nyström Method
• 𝐒 (𝑛×𝑐):columnselectionmatrix• 𝐂 = 𝐊𝐒 (𝑛×𝑐),𝐖 = 𝐒J𝐊𝐒 = 𝐒J𝐂 (𝑐×𝑐)• TheNyström method:𝐊 ≈ 𝐂𝐖7𝐂J
• Newexplanation:• Recallthefastmodel:𝐗n = argmin
𝐗𝐏J 𝐊 − 𝐂𝐗𝐂J 𝐏
5
6
• Setting𝐏 = 𝐒,then𝐗n = argmin
𝐗𝐒J 𝐊 − 𝐂𝐗𝐂J 𝐒
5
6
= 𝐒J𝐂 7 𝐒J𝐊𝐒 𝐂J𝐒 7
= 𝐖7𝐖𝐖7 = 𝐖7
• TheNystrommethodisspecialinstanceofthefastmodel.
• Itisapproximatesolutiontotheprototypemodel
![Page 88: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/88.jpg)
The Nyström Method
• Cost• Time:𝑂 𝑛𝑐6• Memory:𝑂 𝑛𝑐
![Page 89: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/89.jpg)
The Nyström Method
• Cost• Time:𝑂 𝑛𝑐6• Memory:𝑂 𝑛𝑐
Veryefficient!
![Page 90: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/90.jpg)
The Nyström Method
• Cost• Time:𝑂 𝑛𝑐6• Memory:𝑂 𝑛𝑐
• Errorbound:weak
Veryefficient!
![Page 91: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/91.jpg)
Comparisons
• 𝐂 = 𝐊𝐒 ∈ ℝ&×*,𝐖 = 𝐒J𝐊𝐒 = 𝐒J𝐂 ∈ ℝ*×*
• SPSDmatrixapproximation: 𝐊 ≈ 𝐂𝐔𝐂J
• Theprototypemodel:𝐔 = 𝐂7𝐊 𝐂7 J
• Thefastmodel: 𝐔 = 𝐏J𝐂 7(𝐏J𝐊𝐏) 𝐂J𝐏 7
• TheNyström method:𝐔 = 𝐖7
![Page 92: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/92.jpg)
Comparisons
• 𝐂 = 𝐊𝐒 ∈ ℝ&×*,𝐖 = 𝐒J𝐊𝐒 = 𝐒J𝐂 ∈ ℝ*×*
• SPSDmatrixapproximation: 𝐊 ≈ 𝐂𝐔𝐂J
• Theprototypemodel:𝐔 = 𝐂7𝐊 𝐂7 J
• Thefastmodel: 𝐔 = 𝐏J𝐂 7(𝐏J𝐊𝐏) 𝐂J𝐏 7
• TheNyström method:𝐔 = 𝐖7
When 𝐏 = 𝐈&,theprototypemodel⇔ thefastmodel
![Page 93: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/93.jpg)
Comparisons
• 𝐂 = 𝐊𝐒 ∈ ℝ&×*,𝐖 = 𝐒J𝐊𝐒 = 𝐒J𝐂 ∈ ℝ*×*
• SPSDmatrixapproximation: 𝐊 ≈ 𝐂𝐔𝐂J
• Theprototypemodel:𝐔 = 𝐂7𝐊 𝐂7 J
• Thefastmodel:𝐔 = 𝐏J𝐂 7(𝐏J𝐊𝐏) 𝐂J𝐏 7
• TheNyström method:𝐔 = 𝐖7
When 𝐏 = 𝐒,theNyström method⇔ thefastmodel
![Page 94: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/94.jpg)
Comparisons
• 𝑐 = 150,𝑛 = 100𝑐,vary𝑝 from2𝑐 to40𝑐
𝑝/𝑐
𝐊− 𝐂𝐔𝐂J]
6
𝐊 ]6
TheNyströmMethod𝑂 𝑛𝑐6 time
TheFastModel𝑂 𝑛𝑐6 + 𝑝6𝑐 time
ThePrototypeModel𝑂 𝑛6𝑐 time
![Page 95: Randomized SVD, CURDecomposition, and SPSD Matrix …wangshusen.github.io/slides/2016-Matrix-Approximations.pdf · 2020-05-04 · Motivation 2: Matrix Inversion • Solve the linear](https://reader030.vdocuments.us/reader030/viewer/2022041112/5f17a6616ab8435cc65833cc/html5/thumbnails/95.jpg)
Conclusions
• Motivations• Avoidformingthekernelmatrix• Avoidinversion/decomposition
• Prototypemodel,fastmodel,Nystrom• Theyhaveconnections• ThefastmodelandNystromarepractical