CSE6643/MATH6643:Numerical Linear Algebra

Haesun Park
hpark@cc.gatech.edu

School of Computational Science and Engineering

College of Computing
Georgia Institute of Technology
Atlanta, GA 30332, USA

Lecture 2

Introduction
1

1.1

Introduction

Topics

1. Linear system solving Ax = b

2. Least square problem Ax b min kAx bk2
x

3. Eigenvalue problem Ax = x, (x 6= 0)

4. Linear programming min f (x) s.t. Ax = b where f (x) is a linear function.

CSE6643/MATH6643:Numerical Linear Algebra p.1/18

Introduction
1.2

Sherman-Morrison formula

Sherman-Morrison formula:
A is a nonsingular, and v T A1 u 6= 1 then
T 1

(A + uv )

=A

A1 uv T A1

1 + v T A1 u

where we call uv T rank 1 update.

Proof:

T
1
T 1
A1 uv T A1
Iuv T A1
uv T A1 uv T A1
(A+uv ) A 1+vT A1 u = I 1+v
+uv
A

=
T A1 u
1+v T A1 u
I

uv T A1 +(v T A1 u)uv T A1
1+v T A1 u

+ uv T A1 = I

CSE6643/MATH6643:Numerical Linear Algebra p.2/18

Introduction
Sherman-Morrison-Woodbury formula:
A is nonsingular and A Rnn . U Rnk , V Rnk and Ik + V T A1 U is
nonsingular, then
(A + U V T )1 = A1 A1 U (Ik + V T A1 U )1 V A1
0
1
0
1
1 2 3
1
B
C
B
C
Example: A = @ 4 5 6 A, b = @ 1 A
7 8 10
1
Assume
to 3,0i.e. 1
0 A(1, 1) changes
1
3 2 3
2

B
C
B
C
A = @ 4 5 6 A = A + @ 0 A 1 0 0
7 8 10
0

1 uT vA1
1
1
A
b
then x
= A b = A
T
1+v Au

CSE6643/MATH6643:Numerical Linear Algebra p.3/18

Introduction
1.3

Misc

Outer product u v T : n n
Inner product: uT v, 1 1
Range(A) or Ran(A)

Space spanned by columns of A

rank(A) = dim(Range(A))
orthogonal vectors u,v: v t u = kvk2 kuk2 cos = 0
orthonormal vectors u,v kuk2 = 1, kvk2 = 1

orthogonal matrix A Rnn satisfies AT A = In :

T

6
A A=6
4

aT1

h
.. 7
7
. 5 a1
aTn

an

= (aTi aj ) = In

CSE6643/MATH6643:Numerical Linear Algebra p.4/18

Introduction
Special matrix:
diagonal matrix
triangle matrix: upper and lower
Hessenberg matrix: upper and lower
Symmetric matrix AT = A
Symmetric positive (semi) definite: if for all v 6= 0, v T Av > 0 (v T Av 0)

CSE6643/MATH6643:Numerical Linear Algebra p.5/18

Norms & Floating-Point Error

2

Norms & Floating-Point Error

2.1

Vector norm

Definition: Vector norm is a function f : Rn R s.t. f (x) = kxk?

1. kxk 0 and kxk = 0 iff x = 0

2. kxk = || kxk for all R
3. kx + yk kxk + kyk

Common vector norms

`Pn
p 1/p
|x
|
p-norms kxkp =
Pn i=1 i
p = 1, kxk1 = i=1 |xi |
qP
2
n
p = 2, kxk2 =
|x
|
i
i=1
p = , kxk = max1in |xi |
kxk = lim kxkp = lim (|xi |p )1/p = max1in |xi |
p

CSE6643/MATH6643:Numerical Linear Algebra p.6/18

Norms & Floating-Point Error

2.2

Equivalence of Vector Norms

All norms on Rn are equivalent: For two norms kk and kk

there is a constant C > 0 s.t. kxk C kxk for all x Rn .
\

1
2

1
1
1

n
1
1

n
1

CSE6643/MATH6643:Numerical Linear Algebra p.7/18

Norms & Floating-Point Error

2.3

Matrix Norm

Definition: matrix Norm is a function f : Rmn R s.t. f (A) = kAk

1.
2.
3.
4.

kAk 0 and kAk = 0 iff A = 0

kAk = || kAk for all R
kA + Bk kAk + kBk
kABk kAk kBk

CSE6643/MATH6643:Numerical Linear Algebra p.8/18

Norms & Floating-Point Error

Common matrix norms
kAx k

P-norm kAkp = max kxk p for all x Rn1

x6=0
Pm p
kAk1 = max1jn P
i=1 |aij |
|aij |
kAk = max1im n
3
2 j=1
1 2
h
i
6
7
e.g.kAk1 = 12, A = 4 3 4 5 9 12 12
5 6
2
3
2
3
1
2
3
6
7
6
7
e.g.kAk = 11, A = 4 3
4 5 4 7 5 11
5
6
11
kAkp is the p-norm of the largest vector obtained by applying A to a unit
p-norm vector.
Frobenius
norm, NOT one of p-norm
qP
Pm
2
n
|a
|
kAkF =
ij
i=1
j=1
CSE6643/MATH6643:Numerical Linear Algebra p.9/18

Norms & Floating-Point Error

2.4

Equivalence of Matrix Norms

For A Rmn :
\
1
2

1
1
m
m

2
n
1
m

n
n
1

F
n
n
nm
we have kAk2 kAkF
X:n n,non-singular

kAkX = X AX
e.g. kA BkF = 105
Exercise: prove
kAk1 n kAk

kAk1 n kAkF

m
1

n
1

CSE6643/MATH6643:Numerical Linear Algebra p.10/18

Norms & Floating-Point Error

2.5

Orthogonal Transformation Invariant

F-norm and 2-norm are invariant w.r.t. orthogonal transformation

i.e. Q Rmm and Z Rnn are orthogonal matrices
kQAZk2 = kAk2
kQAZkF = kAkF
Will show
1. kQAkF = kAkF

2. kAZkF = kAkF

CSE6643/MATH6643:Numerical Linear Algebra p.11/18

Norms & Floating-Point Error

Proof
h

1. kQAkF = Q a1
kQa1 k22

kQan k22
T

an

i2

2
2
= kQa1 k + + kQan k =
F

+ +
note: kXk22 = X X
kQai k22 = (Qai )T Qai = aTi QT Qai = aTi ai = kai k22
2
3
T

1
6 . 7
7
6
2. exercise kAZkF =
4 .. 5 Z

a
m
F

CSE6643/MATH6643:Numerical Linear Algebra p.12/18

Norms & Floating-Point Error

2.6

Floating-Point Arithmetic

Example. Assume 4-digits of accuracy

f l(1 + 104 1) = f l(f l(1 + 104 ) 1) = 0
f l(1 + 104 1) = f l(f l(1 1) + 104 ) = 104
f l (f l(f l(1 + ) + ) + + ) 6= f l(1 + + f l(f l( + ) + )

Floating-point representation:
Set of floating point numbers
F = (b, t, L, u) = {d1 dt be | 0 dk b 1, d16=0 , L e U } , {0}
where [L, U ] is exponent range, b is the base, t is mantissa length.
For any f F , m |F | M , m = bL1 , M = bu (1 bt ).
overflow |f | > M
underflow |f | < M

CSE6643/MATH6643:Numerical Linear Algebra p.13/18

Norms & Floating-Point Error

Theorem: f l(x) = x(1 + ), || < where is unit round-off , or machine
precision, the smallest number that satisfies f l(1 + ) 6= 1.
e.g. = 2 1016 in matlab
f l(x) = x + x
|f l(x) x| = |x| |x| absolute error
|f l(x)x|
relative error
|x|
e.g. C = AB has two sources of errors
1. representation f l(A) = A + EA , f l(B) = B + EB
2. computation f l(AB) = AB+?
Theorem: A = f l(A) = A + E where E = (eij ), |eij aij |

CSE6643/MATH6643:Numerical Linear Algebra p.14/18

LU Factorization
3

LU Factorization

3.1

Linear Equations

Given A Rnn and b Rn1 . find x such that Ax = b. Solution exists?

1. When A is nonsingular, i.e. det(A) 6= 0, i.e. As columns are linearly

independent , solution unique x = A1 b.
2. When A is singular, b Range(A), infinite solutions.
3. When A is singular, b
/ Range(A), no solution

CSE6643/MATH6643:Numerical Linear Algebra p.15/18

LU Factorization
For what matrix A is Ax = b an easy problem?
1. A is diagonal
2. A is triangular, focus on upper-triangular here
"
#"
# "
#
a11 a12
x1
b1
e.g. n = 2,
=
0
a12
x2
b2
a22 x2 = b2 x2 = b2 /a22
consider general n, k-th equation is
0 + + 0 + akk xk + ak,k+1 xk+1 + + ak,n xn = bk
n
P
xk = (bk
akj xj )/akk
j=k+1

CSE6643/MATH6643:Numerical Linear Algebra p.16/18

LU Factorization
Algorithm 1: Back Substitution
P
2
for k = n : 1 : 1 > n
(2(n

k)
+
1)

n
flops
k=1
for j = k + 1 : n -> 2(n k) flops
bk = bk akj xj
end
xk = bk /ak,k
end

CSE6643/MATH6643:Numerical Linear Algebra p.17/18

LU Factorization
Flops
1 flop one (, /, +, )
(old version) 1 flop one (or/, +or)
What about general A?

6
6
Lets assume A can be factored into A = LU where L = 6
4

and U is upper-triangular.
Then Ax = b can be solved by Ax = b L(U x) = b

1. Solve Ly = b for y
2. Solve U x = y for x

CSE6643/MATH6643:Numerical Linear Algebra p.18/18

1
x
x
x

1
x
x

1
x

3
1

7
7
7,
5