Sparse Matrices

! Have small number of non-zero entries

Dense Matrix

Sparse Matrix

Sparse Matrices
! Non-zeros encode connectivity
! Finite-Element Meshes, Hyperlinks, Social Networks,

Structured Mesh

Unstructured Mesh

General Graph

Sparse Solvers
! Solve sparse linear system (A*x = b) ! Variety of direct and iterative methods

Direct Solvers
! Sparse matrix factorization
! Try to reduce fill-in of factors ! Use sophisticated reordering schemes

! Popular codes: PARDISO, UMFPACK, SuperLU,

Iterative Solvers
! Sequence of approximate solutions
! Converge to exact solution

! Measure error through residual

! residual = b A * x ! residual = A * (x - x) = A * error ! Stop when || b A* x || < tolerance

! Wide variety of methods

! ! ! ! Jacobi Gauss-Seidel Conjugate-Gradient (CG) Generalized Minimum-Residual (GMRES)

Comparison
Direct Solvers ! ! ! ! ! Robust Black-box operation Difficult to parallelize Memory consumption Limited scalability Iterative Solvers ! ! ! ! ! Breakdown issues Lots of parameters Amenable to parallelism Low memory footprint Scalable

Example: Richardson Iteration

r = b while x = r =

A * x norm(r) > tol x + omega * r b A * x

Iterative Solver Components

! Sparse Matrix-Vector Multiplication (SpMV)
! y = A * x

! Vector Scale and Add (SAXPY)

! x = x + omega * r

! Vector Norm (SNRM2)

! norm(r)

Sparse Matrix Storage Formats

! Coordinate (COO)

1 0 5 0

7 2 0 6

0 8 3 0

0 0 9 4

0 0 1

0 1 7

1 1 2

1 2 8

2 0 5

2 2 3

2 3 9

3 1 6

3 row indices 3 4 column indices values

Matrix # entries

10

Sparse Matrix Storage Formats

! Compressed Sparse Row (CSR)
# rows + 1

1 0 5 0

7 2 0 6

0 8 3 0

0 0 9 4

0 0 1

2 1 7

4 1 2

7 2 8

9 0 5 2 3 3 9 1 6 3 4

row offsets column indices values

Matrix # entries
11

Sparse Matrix Storage Formats

! ELLPACK (ELL)
# entries per row

padding 1 2 5 6 7 8 3 4 * * 9 * # rows

1 0 5 0

7 2 0 6

0 8 3 0

0 0 9 4

0 1 0 1

1 2 1 3

* * 2 *

Matrix

column indices

values

12

Sparse Matrix Storage Formats

! Diagonal (DIA)
# diagonals

1 0 5 0

7 2 0 6

0 8 3 0

0 0 9 4

* * 5 6

1 2 3 4

7 8 9 * # rows

Matrix

values

13

Sparse Matrix Storage Formats

! Hybrid (HYB) ELL + COO
0 1 ELL 1 0 5 0 7 2 0 6 0 8 3 0 0 0 9 4 COO 2 3 9 row indices column indices values
14

1 2 1 3

1 2 5 6

7 8 3 4

0 1

column indices

values

Matrix

Storage Format Comparison

Structured

Unstructured

15

Storage Format Comparison

! Structured Mesh

16

Storage Format Comparison

Format COO CSR DIA ELL HYB

float 12.00 8.45 4.05 8.11 8.11

double 16.00 12.45 8.10 12.16 12.16

Matrix

Bytes per Nonzero Entry

17

Storage Format Comparison

! Unstructured Mesh

18

Storage Format Comparison

Format COO CSR DIA ELL HYB

float 12.00 8.62 164.11 11.06 9.00

double 16.00 12.62 328.22 16.60 13.44

Matrix

Bytes per Nonzero Entry

19

Storage Format Comparison

! Random matrix

20

Storage Format Comparison

Format COO CSR DIA ELL HYB

Float 12.00 8.42 76.83 14.20 9.60

Double 16.00 12.42 153.65 21.29 14.20

Matrix

Bytes per Nonzero Entry

21

Storage Format Comparison

! Power-Law Graph

22

Storage Format Comparison

Format COO CSR DIA ELL HYB

Float 12.00 8.74 118.83 49.88 13.50

Double 16.00 12.73 237.66 74.82 19.46

Matrix

Bytes per Nonzero Entry

23

Summary
! Iterative Solvers
! Measure accuracy through residual (b A*x) ! Stop when ||b A*x|| is small

! Sparse Formats
! Numerous Options ! Best format depends on matrix structure

! Next Lesson
! Implementing Sparse Matrix-Vector Multiplication

24

References
Implementing Sparse Matrix-Vector Multiplication on Throughput-Oriented Processors Nathan Bell and Michael Garland Proceedings of Supercomputing '09 Efficient Sparse Matrix-Vector Multiplication on CUDA Nathan Bell and Michael Garland NVIDIA Technical Report NVR-2008-004, December 2008 Iterative Methods for Sparse Linear Systems Yousef Saad http://www-users.cs.umn.edu/~saad/IterMethBook_2ndEd.pdf (online)

25