book 2006/ page

Contents
Preface iii

I 1

Linear Algebra Concepts and Matrix Decompositions Vectors and Matrices in Data Mining and Pattern 1.1 Data Mining and Pattern Recognition . . . . . . . 1.2 Vectors and Matrices . . . . . . . . . . . . . . . . 1.3 Purpose of the Book . . . . . . . . . . . . . . . . 1.4 Programming Environments . . . . . . . . . . . . 1.5 Floating Point Computations . . . . . . . . . . . . 1.6 Notation and Conventions . . . . . . . . . . . . . Vectors and Matrices 2.1 Matrix-Vector Multiplication . . . 2.2 Matrix-Matrix Multiplication . . 2.3 Scalar Product and Vector Norms 2.4 Matrix Norms . . . . . . . . . . . 2.5 Linear Independence Bases . . . 2.6 The Rank of a Matrix . . . . . . . Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1 1 1 2 5 6 6 9 11 11 13 15 16 17 18 21 21 23 24 25 26 28 35 36 38 43 44

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

Linear Systems and Least Squares 3.1 LU Decomposition . . . . . . . . . . . . . . . 3.2 Symmetric, Positive Denite Matrices . . . . 3.3 Perturbation Theory and Condition Number 3.4 Rounding Errors in Gaussian Elimination . . 3.5 Banded Matrices . . . . . . . . . . . . . . . . 3.6 The Least Squares Problem . . . . . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

Orthogonality 4.1 Orthogonal Vectors and Matrices . . . . . . . . . . . . . . 4.2 Elementary Orthogonal Matrices . . . . . . . . . . . . . . . 4.3 Number of Floating Point Operations . . . . . . . . . . . . 4.4 Orthogonal Transformations in Floating Point Arithmetic v

. . . .

. . . .

. . . .

book 2006/ page v

vi 5 QR Decomposition 5.1 Orthogonal Transformation to Triangular Form . . 5.2 Solving the Least Squares Problem . . . . . . . . . 5.3 Computing or Not Computing Q . . . . . . . . . . 5.4 Flop Count for QR Factorization . . . . . . . . . . 5.5 Error in the Solution of the Least Squares Problem 5.6 Updating the Solution of a Least Squares Problem .

Contents 45 45 49 50 51 51 52

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

. . . . . .

Singular Value Decomposition 55 6.1 The Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . 55 6.2 Fundamental Subspaces . . . . . . . . . . . . . . . . . . . . . . . 59 6.3 Matrix Approximation . . . . . . . . . . . . . . . . . . . . . . . 61 6.4 Principal Component Analysis . . . . . . . . . . . . . . . . . . . 64 6.5 Solving Least Squares Problems . . . . . . . . . . . . . . . . . . 65 6.6 Condition Number and Perturbation Theory for the Least Squares Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 6.7 Rank-Decient and Under-Determined Systems . . . . . . . . . 68 6.8 Computing the SVD . . . . . . . . . . . . . . . . . . . . . . . . 70 6.9 Complete Orthogonal Decomposition . . . . . . . . . . . . . . . 70 Reduced Rank Least Squares Models 7.1 Truncated SVD: Principal Components Regression . . . . . . . 7.2 A Krylov Subspace Method . . . . . . . . . . . . . . . . . . . . Tensor Decomposition 8.1 Introduction . . . . . . . . . . . . . . . . 8.2 Basic Tensor Concepts . . . . . . . . . . 8.3 A Tensor Singular Value Decomposition . 8.4 Approximating a Tensor by HOSVD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 75 78 89 89 90 92 94

Clustering and Non-Negative Matrix Factorization 99 9.1 The k -Means Algorithm . . . . . . . . . . . . . . . . . . . . . . 100 9.2 Non-Negative Matrix Factorization . . . . . . . . . . . . . . . . 104 Data Mining Applications 109

II 10

Classication of Handwritten Digits 111 10.1 Handwritten Digits and a Simple Algorithm . . . . . . . . . . . 111 10.2 Classication using SVD Bases . . . . . . . . . . . . . . . . . . . 113 10.3 Tangent Distance . . . . . . . . . . . . . . . . . . . . . . . . . . 118 Text Mining 11.1 Preprocessing the Documents and Queries 11.2 The Vector Space Model . . . . . . . . . . 11.3 Latent Semantic Indexing . . . . . . . . . . 11.4 Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 128 129 132 137

11

book 2006/ page v

Contents 11.5 11.6 11.7 12 Page 12.1 12.2 12.3 12.4

vii Non-Negative Matrix Factorization . . . . . . . . . . . . . . . . 138 Lanczos-Golub-Kahan Bidiagonalization . . . . . . . . . . . . . 140 Average Performance . . . . . . . . . . . . . . . . . . . . . . . . 143 Ranking for a Web Search Engine Pagerank . . . . . . . . . . . . . . . . . . . . . Random Walk and Markov Chains . . . . . . . The Power Method for Pagerank Computation HITS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 145 148 152 157

13

Automatic Key Word and Key Sentence Extraction 159 13.1 Saliency Score . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 13.2 Key Sentence Extraction from a Rank-k Approximation . . . . . 163 Face Recognition Using Tensor SVD 167 14.1 Tensor Representation . . . . . . . . . . . . . . . . . . . . . . . 167 14.2 Face Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . 170 14.3 Face Recognition with HOSVD Compression . . . . . . . . . . . 173 175 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 178 183 185 187 194 195 197 198 205 207 215

14

III Computing the Matrix Decompositions 15 Computing Eigenvalues and Singular Values 15.1 Perturbation Theory . . . . . . . . . . . . . . . . . . . 15.2 The Power Method and Inverse Iteration . . . . . . . . 15.3 Similarity Reduction to Tridiagonal Form . . . . . . . . 15.4 The QR Algorithm for a Symmetric Tridiagonal Matrix 15.5 Computing the SVD . . . . . . . . . . . . . . . . . . . 15.6 The Non-Symmetric Eigenvalue Problem . . . . . . . . 15.7 Sparse Matrices . . . . . . . . . . . . . . . . . . . . . . 15.8 The Arnoldi and Lanczos Methods . . . . . . . . . . . 15.9 Software . . . . . . . . . . . . . . . . . . . . . . . . . .

Bibliography Index