![Page 1: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/1.jpg)
![Page 2: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/2.jpg)
2
Sparse Matrices
! Have small number of non-zero entries
Dense Matrix Sparse Matrix
![Page 3: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/3.jpg)
3
Sparse Matrices
! Non-zeros encode connectivity ! Finite-Element Meshes, Hyperlinks, Social Networks, …
Structured Mesh Unstructured Mesh General Graph
![Page 4: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/4.jpg)
4
Sparse Solvers
! Solve sparse linear system (A*x = b) ! Variety of direct and iterative methods
![Page 5: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/5.jpg)
5
Direct Solvers
! Sparse matrix factorization ! Try to reduce fill-in of factors ! Use sophisticated reordering schemes
! Popular codes: PARDISO, UMFPACK, SuperLU, …
![Page 6: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/6.jpg)
6
Iterative Solvers
! Sequence of approximate solutions ! Converge to exact solution
! Measure error through residual ! residual = b – A * x ! residual = A * (x’ - x) = A * error ! Stop when || b – A* x || < tolerance
! Wide variety of methods ! Jacobi ! Gauss-Seidel ! Conjugate-Gradient (CG) ! Generalized Minimum-Residual (GMRES)
![Page 7: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/7.jpg)
7
Comparison
Direct Solvers ! Robust ! Black-box operation ! Difficult to parallelize ! Memory consumption ! Limited scalability
Iterative Solvers ! Breakdown issues ! Lots of parameters ! Amenable to parallelism ! Low memory footprint ! Scalable
![Page 8: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/8.jpg)
8
Example: Richardson Iteration
r = b – A * x while norm(r) > tol x = x + omega * r r = b – A * x
![Page 9: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/9.jpg)
9
Iterative Solver Components
! Sparse Matrix-Vector Multiplication (SpMV) ! y = A * x
! Vector Scale and Add (SAXPY) ! x = x + omega * r
! Vector Norm (SNRM2) ! norm(r)
![Page 10: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/10.jpg)
10
Sparse Matrix Storage Formats
! Coordinate (COO)
1 7 0 0 0 2 8 0 5 0 3 9 0 6 0 4
row indices
column indices
values
Matrix
0 1 1 2 0 2 3 1 3
1 7 2 8 5 3 9 6 4
0 0 1 1 2 2 2 3 3
# entries
![Page 11: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/11.jpg)
11
Sparse Matrix Storage Formats
! Compressed Sparse Row (CSR)
1 7 0 0 0 2 8 0 5 0 3 9 0 6 0 4
row offsets
column indices
values
Matrix
0 1 1 2 0 2 3 1 3
1 7 2 8 5 3 9 6 4
0 2 4 7 9
# entries
# rows + 1
![Page 12: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/12.jpg)
12
Sparse Matrix Storage Formats
! ELLPACK (ELL)
1 7 0 0 0 2 8 0 5 0 3 9 0 6 0 4
column indices values Matrix
# rows
0 1 * 1 2 * 0 1 2 1 3 *
1 7 * 2 8 * 5 3 9 6 4 *
# entries per row padding
![Page 13: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/13.jpg)
13
Sparse Matrix Storage Formats
! Diagonal (DIA)
1 7 0 0 0 2 8 0 5 0 3 9 0 6 0 4
values Matrix
# rows
* 1 7 * 2 8 5 3 9 6 4 *
# diagonals
![Page 14: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/14.jpg)
14
Sparse Matrix Storage Formats
! Hybrid (HYB) ELL + COO
1 7 0 0 0 2 8 0 5 0 3 9 0 6 0 4
column indices values
Matrix
0 1 1 2 0 1 1 3
1 7 2 8 5 3 6 4
row indices
column indices
values
3
9
2
ELL
COO
![Page 15: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/15.jpg)
15
Storage Format Comparison
Structured Unstructured
![Page 16: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/16.jpg)
16
Storage Format Comparison
! Structured Mesh
![Page 17: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/17.jpg)
17
Storage Format Comparison
Matrix
Format float double
COO 12.00 16.00
CSR 8.45 12.45
DIA 4.05 8.10
ELL 8.11 12.16
HYB 8.11 12.16
Bytes per Nonzero Entry
![Page 18: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/18.jpg)
18
Storage Format Comparison
! Unstructured Mesh
![Page 19: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/19.jpg)
19
Storage Format Comparison
Format float double
COO 12.00 16.00
CSR 8.62 12.62
DIA 164.11 328.22
ELL 11.06 16.60
HYB 9.00 13.44
Matrix Bytes per Nonzero Entry
![Page 20: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/20.jpg)
20
Storage Format Comparison
! Random matrix
![Page 21: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/21.jpg)
21
Storage Format Comparison
Format Float Double
COO 12.00 16.00
CSR 8.42 12.42
DIA 76.83 153.65
ELL 14.20 21.29
HYB 9.60 14.20
Matrix Bytes per Nonzero Entry
![Page 22: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/22.jpg)
22
Storage Format Comparison
! Power-Law Graph
![Page 23: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/23.jpg)
23
Storage Format Comparison
Format Float Double
COO 12.00 16.00
CSR 8.74 12.73
DIA 118.83 237.66
ELL 49.88 74.82
HYB 13.50 19.46
Matrix Bytes per Nonzero Entry
![Page 24: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/24.jpg)
24
Summary
! Iterative Solvers ! Measure accuracy through residual (b – A*x) ! Stop when ||b – A*x|| is small
! Sparse Formats ! Numerous Options ! Best format depends on matrix structure
! Next Lesson ! Implementing Sparse Matrix-Vector Multiplication
![Page 25: Sparse Matrices - Boston University Sparse Matrices ! Have small number of non-zero entries Dense Matrix Sparse Matrix](https://reader033.vdocuments.us/reader033/viewer/2022051407/5ae6aa217f8b9ae1578dff35/html5/thumbnails/25.jpg)
25
References
Implementing Sparse Matrix-Vector Multiplication on Throughput-Oriented Processors Nathan Bell and Michael Garland Proceedings of Supercomputing '09
Efficient Sparse Matrix-Vector Multiplication on CUDA Nathan Bell and Michael Garland NVIDIA Technical Report NVR-2008-004, December 2008
Iterative Methods for Sparse Linear Systems Yousef Saad http://www-users.cs.umn.edu/~saad/IterMethBook_2ndEd.pdf (online)