randomized kaczmarz - blogsrandomized kaczmarz iterative algorithm for solving exponential...
TRANSCRIPT
![Page 1: Randomized Kaczmarz - BlogsRandomized Kaczmarz Iterative algorithm for solving Exponential convergence in m.s. (SV’09, FZ’12) • Rate of convergence: Projection to the solution](https://reader033.vdocuments.us/reader033/viewer/2022041708/5e460324723d155eef59afd7/html5/thumbnails/1.jpg)
Randomized Kaczmarz
Nick Freris LCAV
March 7, 2013
![Page 2: Randomized Kaczmarz - BlogsRandomized Kaczmarz Iterative algorithm for solving Exponential convergence in m.s. (SV’09, FZ’12) • Rate of convergence: Projection to the solution](https://reader033.vdocuments.us/reader033/viewer/2022041708/5e460324723d155eef59afd7/html5/thumbnails/2.jpg)
Outline
▪ Randomized Kaczmarz algorithm • Consistent systems (noiseless) • Inconsistent systems (noisy)
▪ Optimal de-noising • Convergence analysis and simulations
▪ Application in sensor networks • Distributed consensus algorithm for synchronization
▪ Faster convergence and energy savings
1) Speed-up for sparse systems
2) Novel consensus design method
1 / 13
![Page 3: Randomized Kaczmarz - BlogsRandomized Kaczmarz Iterative algorithm for solving Exponential convergence in m.s. (SV’09, FZ’12) • Rate of convergence: Projection to the solution](https://reader033.vdocuments.us/reader033/viewer/2022041708/5e460324723d155eef59afd7/html5/thumbnails/3.jpg)
Randomized Kaczmarz ▪ Iterative algorithm for solving
▪ Exponential convergence in m.s. (SV’09, FZ’12) • Rate of convergence:
Projection to the solution space of selected row
Randomized selection of row
2 / 13
![Page 4: Randomized Kaczmarz - BlogsRandomized Kaczmarz Iterative algorithm for solving Exponential convergence in m.s. (SV’09, FZ’12) • Rate of convergence: Projection to the solution](https://reader033.vdocuments.us/reader033/viewer/2022041708/5e460324723d155eef59afd7/html5/thumbnails/4.jpg)
Noisy case ▪ Noisy measurements: ▪ Oscillatory behavior
• Asymptotically constrained in a ball (N’10, FZ’12)
▪ Under-relaxation (RKU)
• Convergence to a point in the ball ▪ slower
▪ Least-squares: • Bad idea (doubling the condition number)
3 / 13
![Page 5: Randomized Kaczmarz - BlogsRandomized Kaczmarz Iterative algorithm for solving Exponential convergence in m.s. (SV’09, FZ’12) • Rate of convergence: Projection to the solution](https://reader033.vdocuments.us/reader033/viewer/2022041708/5e460324723d155eef59afd7/html5/thumbnails/5.jpg)
Optimal de-noising ▪ LS for inconsistent system:
• Solution: projection to the range space of A
• same rate of convergence
Convergence (ZF’13):
Ax = bR(A)
Projection to the orthogonal complement of the selected column
Randomized selection of column
4 / 13
![Page 6: Randomized Kaczmarz - BlogsRandomized Kaczmarz Iterative algorithm for solving Exponential convergence in m.s. (SV’09, FZ’12) • Rate of convergence: Projection to the solution](https://reader033.vdocuments.us/reader033/viewer/2022041708/5e460324723d155eef59afd7/html5/thumbnails/6.jpg)
Putting the pieces together
Randomized orthogonal projection
Randomized Kaczmarz
▪ RK and de-noising:
▪ Convergence (ZF’13):
• same exponent, no delay • expected number of arithmetic operations:
Ekx(k) � xLSk2 (1� 1
2F (A)
)[kxLSk2 + ckbR(A)k2k]
Proportional to sparsity
5 / 13
![Page 7: Randomized Kaczmarz - BlogsRandomized Kaczmarz Iterative algorithm for solving Exponential convergence in m.s. (SV’09, FZ’12) • Rate of convergence: Projection to the solution](https://reader033.vdocuments.us/reader033/viewer/2022041708/5e460324723d155eef59afd7/html5/thumbnails/7.jpg)
Experiments
Excellent performance in sparse systems
6 / 13
![Page 8: Randomized Kaczmarz - BlogsRandomized Kaczmarz Iterative algorithm for solving Exponential convergence in m.s. (SV’09, FZ’12) • Rate of convergence: Projection to the solution](https://reader033.vdocuments.us/reader033/viewer/2022041708/5e460324723d155eef59afd7/html5/thumbnails/8.jpg)
A sensor network problem
▪ Relative measurements • For two neighbors: • Network problem:
▪ Jacobi algorithm for LSE
• Local averaging (distributed)
• Synchronous: Exponential convergence (GK’06) • Asynchronous: Exponential convergence (FZ’12)
▪ Applications • Clock synchronization • Localization
7 / 13
![Page 9: Randomized Kaczmarz - BlogsRandomized Kaczmarz Iterative algorithm for solving Exponential convergence in m.s. (SV’09, FZ’12) • Rate of convergence: Projection to the solution](https://reader033.vdocuments.us/reader033/viewer/2022041708/5e460324723d155eef59afd7/html5/thumbnails/9.jpg)
Smoothing via RK
▪ Asynchronous implementation • Exponential clocks
Distributed averaging
8 / 13
![Page 10: Randomized Kaczmarz - BlogsRandomized Kaczmarz Iterative algorithm for solving Exponential convergence in m.s. (SV’09, FZ’12) • Rate of convergence: Projection to the solution](https://reader033.vdocuments.us/reader033/viewer/2022041708/5e460324723d155eef59afd7/html5/thumbnails/10.jpg)
Convergence analysis
Algorithm Convergence Reference
Jacobi GK’06 (FZ’12)
OSE Faster than Jacobi BDE’06
RKS FZ’12
RKLS FZ’12
RKU FZ’12
RKO Faster than RKS FZ’12
▪ Cheeger’s inequality:
9 / 13
![Page 11: Randomized Kaczmarz - BlogsRandomized Kaczmarz Iterative algorithm for solving Exponential convergence in m.s. (SV’09, FZ’12) • Rate of convergence: Projection to the solution](https://reader033.vdocuments.us/reader033/viewer/2022041708/5e460324723d155eef59afd7/html5/thumbnails/11.jpg)
An extension
▪ “Over-smoothing” (RKO)
•
• Faster convergence in absolute time vs
• More messages exchanged per iteration
10 / 13
![Page 12: Randomized Kaczmarz - BlogsRandomized Kaczmarz Iterative algorithm for solving Exponential convergence in m.s. (SV’09, FZ’12) • Rate of convergence: Projection to the solution](https://reader033.vdocuments.us/reader033/viewer/2022041708/5e460324723d155eef59afd7/html5/thumbnails/12.jpg)
Simulations
Faster convergence Energy savings
11 / 13
![Page 13: Randomized Kaczmarz - BlogsRandomized Kaczmarz Iterative algorithm for solving Exponential convergence in m.s. (SV’09, FZ’12) • Rate of convergence: Projection to the solution](https://reader033.vdocuments.us/reader033/viewer/2022041708/5e460324723d155eef59afd7/html5/thumbnails/13.jpg)
Conclusions ▪ Randomized Kaczmarz (RK) algorithm
• Exponential convergence in the mean-square ▪ Same rate regardless of noise
• Distributed asynchronous smoothing
▪ Experiments • Linear systems: Gains for sparse systems • Sensor networks: Faster convergence and energy savings
Efficient for sparse systems
12 / 13
![Page 14: Randomized Kaczmarz - BlogsRandomized Kaczmarz Iterative algorithm for solving Exponential convergence in m.s. (SV’09, FZ’12) • Rate of convergence: Projection to the solution](https://reader033.vdocuments.us/reader033/viewer/2022041708/5e460324723d155eef59afd7/html5/thumbnails/14.jpg)
Future work
▪ Distributed implementation of REK • de-noising • matrix pre-conditioning • termination criteria
▪ Stochastic approximation • convergence to the true values
▪ slower (gradient method) • improved convergence
13 / 13
Numerical analysis is not dead!
![Page 15: Randomized Kaczmarz - BlogsRandomized Kaczmarz Iterative algorithm for solving Exponential convergence in m.s. (SV’09, FZ’12) • Rate of convergence: Projection to the solution](https://reader033.vdocuments.us/reader033/viewer/2022041708/5e460324723d155eef59afd7/html5/thumbnails/15.jpg)
References
1. N. Freris and A. Zouzias, “Fast distributed smoothing of relative measurements," 51st
IEEE Conference on Decision and Control (CDC), pp.1411-1416, 10-13 Dec. 2012.
2. A. Zouzias and N. Freris, “Randomized Extended Kaczmarz for Solving Least Squares,” accepted in SIAM Journal on Matrix Analysis and Applications (SIMAX)
Available: http://arxiv.org/abs/1205.5770 3. T. Strohmer and R. Vershynin, “A Randomized Kaczmarz Algorithm with Exponential
Convergence,” Journal of Fourier Analysis and Applications, vol. 15, no. 1, pp. 262–278, 2009.
4. D. Needell. “Randomized Kaczmarz Solver for Noisy Linear Systems.” Bit Numerical Mathematics, 50(2):395–403, 2010.
![Page 16: Randomized Kaczmarz - BlogsRandomized Kaczmarz Iterative algorithm for solving Exponential convergence in m.s. (SV’09, FZ’12) • Rate of convergence: Projection to the solution](https://reader033.vdocuments.us/reader033/viewer/2022041708/5e460324723d155eef59afd7/html5/thumbnails/16.jpg)
Thank you