The Matrix Singular Value Decomposition

Singular Value Decomposition(SVD)

The Singular Value Decomposition (SVD) of a matrix A ∈ Rn×m is a

decomposition of the form

A = UΣV T

where U ∈ Rn×n and V ∈ Rm×m are orthogonal matrices and

Σ := diag(σ1 , σ2 , . . . σp ) ∈ Rn×m is a diagonal matrix with

σ1 ≥ σ2 ≥ · · · σp ≥ 0

for p = min{n, m}.

Singular Value Decomposition(SVD)

The Singular Value Decomposition (SVD) of a matrix A ∈ Rn×m is a

decomposition of the form

A = UΣV T

where U ∈ Rn×n and V ∈ Rm×m are orthogonal matrices and

Σ := diag(σ1 , σ2 , . . . σp ) ∈ Rn×m is a diagonal matrix with

σ1 ≥ σ2 ≥ · · · σp ≥ 0

for p = min{n, m}.

If A is a complex n × m matrix, then the above decomposition holds
with T replaced by ∗ so that U and V are unitary matrices. However
Σ remains unchanged.
Singular Value Decomposition(SVD)

The Singular Value Decomposition (SVD) of a matrix A ∈ Rn×m is a

decomposition of the form

A = UΣV T

where U ∈ Rn×n and V ∈ Rm×m are orthogonal matrices and

Σ := diag(σ1 , σ2 , . . . σp ) ∈ Rn×m is a diagonal matrix with

σ1 ≥ σ2 ≥ · · · σp ≥ 0

for p = min{n, m}.

If A is a complex n × m matrix, then the above decomposition holds
with T replaced by ∗ so that U and V are unitary matrices. However
Σ remains unchanged.
The numbers σ1 , σ2 , . . . , σp are called the singular values of A.
Singular Value Decomposition(SVD)
Every matrix has an SVD. For example, the SVD of
" # " #T
√1 √1 √1 − √12
 
1 2 2 2 3 0 2
A := = ,
2 1 √1 − √12 0 1 √1 √1
2 2 2
Singular Value Decomposition(SVD)
Every matrix has an SVD. For example, the SVD of
" # " #T
√1 √1 √1 − √12
 
1 2 2 2 3 0 2
A := = ,
2 1 √1 − √12 0 1 √1 √1
2 2 2

" #T
   √  √1 − √25
B := 1 2 = [1] 5 ,
5 0 √2 √1
5 5
Singular Value Decomposition(SVD)
Every matrix has an SVD. For example, the SVD of
" # " #T
√1 √1 √1 − √12
 
1 2 2 2 3 0 2
A := = ,
2 1 √1 − √12 0 1 √1 √1
2 2 2

" #T
   √  √1 − √25
B := 1 2 = [1] 5 ,
5 0 √2 √1
5 5

" # √
√1 − √12
  
1 2
D := = 2 [1]T .
1 √1 √1 0
2 2
Singular Value Decomposition(SVD)
Every matrix has an SVD. For example, the SVD of
" # " #T
√1 √1 √1 − √12
 
1 2 2 2 3 0 2
A := = ,
2 1 √1 − √12 0 1 √1 √1
2 2 2

" #T
   √  √1 − √25
B := 1 2 = [1] 5 ,
5 0 √2 √1
5 5

" # √
√1 − √12
  
1 2
D := = 2 [1]T .
1 √1 √1 0
2 2

Clearly if A = Udiag(σ1 , σ2 , . . . σp )V T is the SVD of A and rank A = r ,

then the first r singular values σ1 ≥ · · · ≥ σr > 0 with σk = 0 for
k = r + 1, . . . , p if r < p.
Singular Value Decomposition(SVD)
If U = [u1 · · · un ] and V = [v1 · · · vm ] , then for i = 1, . . . , p,
Avi = σi ui and ui∗ A = σi vi∗
Hence ui and vi are respectively left and right singular vectors of A
corresponding to σi .
Theorem Every matrix has a Singular Value Decomposition.
Theorem Let A = UΣV ∗ be an SVD of A ∈ Fn×m with rank A = r .
(a) A = Σri=1 σi ui vi∗
(b) A∗ = V ΣT U ∗ is an SVD of A∗ .
(c)
R(A) = span{u1 , . . . , ur }, N(A) = span{vr +1 , . . . , vm }
∗
R(A ) = span{v1 , . . . , vr } N(A∗ ) = span{ur +1 , . . . , un }.

(d) kAk2 = σ1 .
q
(e) kAkF = Σrk =1 σk2 .

(Here F = R or F = C.)
Singular Value Decomposition(SVD)
If U = [u1 · · · un ] and V = [v1 · · · vm ] , then for i = 1, . . . , p,
Avi = σi ui and ui∗ A = σi vi∗
Hence ui and vi are respectively left and right singular vectors of A
corresponding to σi .
Theorem Every matrix has a Singular Value Decomposition.
Theorem Let A = UΣV ∗ be an SVD of A ∈ Fn×m with rank A = r .
(a) A = Σri=1 σi ui vi∗ (Exercise!)
(b) A∗ = V ΣT U ∗ is an SVD of A∗ . (Exercise!)
(c)
R(A) = span{u1 , . . . , ur }, N(A) = span{vr +1 , . . . , vm }
∗
R(A ) = span{v1 , . . . , vr } N(A∗ ) = span{ur +1 , . . . , un }.

(d) kAk2 = σ1 .
q
(e) kAkF = Σrk =1 σk2 .

(Here F = R or F = C.)
Singular Value Decomposition(SVD)
Corollary Let A = UΣV ∗ be an SVD of A ∈ Fn×m .

(a) If A is square and nonsingular, then A−1 = (VF )(F Σ−1 F )(UF )∗
is an SVD of A−1 and where F is the n × n ‘flip’ matrix and
kA−1 k2 = σ1n .
 σ1
σm if n ≥ m and rank A = m.
(b) κ2 (A) =
∞ otherwise
(c) Assuming, σk = 0 for k > min{m, n},
A∗ Avi = σi2 vi , i = 1, . . . , m, and AA∗ uj = σj2 uj , j = 1, . . . , n.
(d) If n = m and A is a singular matrix, then for any  > 0, there
exists a nonsingular matrix B ∈ Fn×n such that kA − Bk2 < .
Singular Value Decomposition(SVD)
Corollary Let A = UΣV ∗ be an SVD of A ∈ Fn×m .

(a) If A is square and nonsingular, then A−1 = (VF )(F Σ−1 F )(UF )∗
is an SVD of A−1 and where F is the n × n ‘flip’ matrix and
kA−1 k2 = σ1n .
 σ1
σm if n ≥ m and rank A = m.
(b) κ2 (A) =
∞ otherwise
(c) Assuming, σk = 0 for k > min{m, n},
A∗ Avi = σi2 vi , i = 1, . . . , m, and AA∗ uj = σj2 uj , j = 1, . . . , n.
(d) If n = m and A is a singular matrix, then for any  > 0, there
exists a nonsingular matrix B ∈ Fn×n such that kA − Bk2 < .

Condensed Singular Value Decomposition: Let A = UΣV ∗ be an

SVD of A ∈ Fn×m with rank A = r . Let U = [u1 u2 · · · ur ] ∈ Fn×r ,
Vr = [v1 v2 · · · vr ] ∈ Fm×r and Σr = diag(σ1 , . . . σr ) ∈ Fr ×r . Then

A = Ur Σr Vr∗

is called the Condensed Singular Value Decomposition of A.

Singular Value Decomposition(SVD)
Corollary Let A = UΣV ∗ be an SVD of A ∈ Fn×m .

(a) If n = m and A is nonsingular, then A−1 = V Σ−1 U ∗ is an SVD of

A−1 and kA−1 k2 = σ1n . (Exercise!)
 σ1
σm if n ≥ m and rank A = m.
(b) κ2 (A) = (Exercise!)
∞ otherwise
(c) Assuming, σk = 0 for k > min{m, n},
A∗ Avi = σi2 vi , i = 1, . . . , m, and AA∗ uj = σj2 uj , j = 1, . . . , n.
(Exercise!)
(d) If n = m and A is a singular matrix, then for any  > 0, there
exists a nonsingular matrix B ∈ Fn×n such that kA − Bk2 < .

Condensed Singular Value Decomposition: Let A = UΣV ∗ be an

SVD of A ∈ Fn×m with rank A = r . Let U = [u1 u2 · · · ur ] ∈ Fn×r ,
Vr = [v1 v2 · · · vr ] ∈ Fm×r and Σr = diag(σ1 , . . . σr ) ∈ Fr ×r . Then

A = Ur Σr Vr∗

is called the Condensed Singular Value Decomposition of A.

Moore-Penrose Pseudoinverse
Let A = UΣV ∗ be an SVD of A ∈ Fn×m with rank A = r . The
Moore-Penrose pseudoinverse A† of A is defined as

A† := V Σ† U ∗

where Σ† = diag(σ1−1 , . . . , σr−1 , 0, . . . 0) ∈ Rn×m .

Moore-Penrose Pseudoinverse
Let A = UΣV ∗ be an SVD of A ∈ Fn×m with rank A = r . The
Moore-Penrose pseudoinverse A† of A is defined as

A† := V Σ† U ∗

where Σ† = diag(σ1−1 , . . . , σr−1 , 0, . . . 0) ∈ Rn×m .

Theorem Let A ∈ Fn×m . Then,

(a) A−1 = A† if n = m and A is nonsingular.

(b) A† = (A∗ A)−1 A∗ if rank A = m
(c) A† = A∗ (AA∗ )−1 if rank A = n.
(d) (AA† )∗ = AA† , (A† A)∗ = A† A, AA† A = A, A† AA† = A† .
Also, if B ∈ Fm×n , such that
(AB)∗ = AB, (BA)∗ = BA, ABA = A, BAB = B, then
B = A† .
(f) (A† )∗ = (A∗ )† .
(g) A† = Vr Σ−1 ∗
r Ur .
Moore-Penrose Pseudoinverse
Let A = UΣV ∗ be an SVD of A ∈ Fn×m with rank A = r . The
Moore-Penrose pseudoinverse A† of A is defined as

A† := V Σ† U ∗

where Σ† = diag(σ1−1 , . . . , σr−1 , 0, . . . 0) ∈ Rn×m .

Theorem Let A ∈ Fn×m . Then,

(a) A−1 = A† if n = m and A is nonsingular. (Exercise!)

(b) A† = (A∗ A)−1 A∗ if rank A = m (Exercise!)
(c) A† = A∗ (AA∗ )−1 if rank A = n. (Exercise!)
(d) (AA† )∗ = AA† , (A† A)∗ = A† A, AA† A = A, A† AA† = A† .
Also, if B ∈ Fm×n , such that
(AB)∗ = AB, (BA)∗ = BA, ABA = A, BAB = B, then
B = A† . (Exercise!)
(f) (A† )∗ = (A∗ )† . (Exercise!)
(g) A† = Vr Σ−1 ∗
r Ur . (Exercise!)
Moore-Penrose Pseudoinverse and the LSP
Moore-Penrose Pseudoinverse and the LSP

Theorem Let Ax = b where A ∈ Rn×m and b ∈ Rn with n ≥ m.

Then x0 = A† b is the unique least squares solution of the
system Ax = b if rank A = m.
If rank A < m, then x0 is the least squares solution of the
system with the smallest 2-norm.
Moore-Penrose Pseudoinverse and the LSP

Theorem Let Ax = b where A ∈ Rn×m and b ∈ Rn with n ≥ m.

Then x0 = A† b is the unique least squares solution of the
system Ax = b if rank A = m.
If rank A < m, then x0 is the least squares solution of the
system with the smallest 2-norm.

Corollary Let Ax = b where A ∈ Rn×m and b ∈ Rn with n ≥ m.

Then x0 = Vr Σ−1 T
r Ur b is the least squares solution of Ax = b of
least 2-norm.
Moore-Penrose Pseudoinverse and the LSP

Theorem Let Ax = b where A ∈ Rn×m and b ∈ Rn with n ≥ m.

Then x0 = A† b is the unique least squares solution of the
system Ax = b if rank A = m.
If rank A < m, then x0 is the least squares solution of the
system with the smallest 2-norm.

Corollary Let Ax = b where A ∈ Rn×m and b ∈ Rn with n ≥ m.

Then x0 = Vr Σ−1 T
r Ur b is the least squares solution of Ax = b of
least 2-norm.
The main flop count of solving the LSP problem associated with
an overdetermined system Ax = b is that of computing the
Condensed SVD of A.
Eckart-Young Theorem
Theorem[Schmidt, 1907], [Eckart & Young, 1936]
Let A ∈ Fn×m with rank A = r . Let A = UΣV ∗ be an SVD of A. For
k = 1, . . . r − 1, define
Ak = UΣk V ∗
where Σk = diag(σ1 , . . . , σk , 0, . . . , 0) ∈ Rn×m is a diagonal matrix.
Then rank Ak = k and

kA − Ak k2 = min{kA − Bk2 : B ∈ Fn×m with rank B ≤ k } = σk +1 .

Eckart-Young Theorem
Theorem[Schmidt, 1907], [Eckart & Young, 1936]
Let A ∈ Fn×m with rank A = r . Let A = UΣV ∗ be an SVD of A. For
k = 1, . . . r − 1, define
Ak = UΣk V ∗
where Σk = diag(σ1 , . . . , σk , 0, . . . , 0) ∈ Rn×m is a diagonal matrix.
Then rank Ak = k and

kA − Ak k2 = min{kA − Bk2 : B ∈ Fn×m with rank B ≤ k } = σk +1 .

Corollary Let A ∈ Fn×n be nonsingular. Let A = UΣV ∗ be an SVD of

A. Then,
σn = min{kA − Bk2 : B ∈ Fn×n is singular}.
(Exercise!)
Eckart-Young Theorem
Theorem[Schmidt, 1907], [Eckart & Young, 1936]
Let A ∈ Fn×m with rank A = r . Let A = UΣV ∗ be an SVD of A. For
k = 1, . . . r − 1, define
Ak = UΣk V ∗
where Σk = diag(σ1 , . . . , σk , 0, . . . , 0) ∈ Rn×m is a diagonal matrix.
Then rank Ak = k and

kA − Ak k2 = min{kA − Bk2 : B ∈ Fn×m with rank B ≤ k } = σk +1 .

Corollary Let A ∈ Fn×n be nonsingular. Let A = UΣV ∗ be an SVD of

A. Then,
σn = min{kA − Bk2 : B ∈ Fn×n is singular}.
(Exercise!)
Corollary Let A ∈ Fn×n be nonsingular. Then,
 
1 k∆Ak2
= min : A + ∆A is singular
κ2 (A) kAk2

(Exercise!)
Numerical rank determination via SVD
Let A = UΣV ∗ be an SVD of an n × m real or complex matrix A with

Σ := diag(σ1 , σ2 , . . . σp ) ∈ Rn×m

where σ1 ≥ σ2 ≥ · · · σp ≥ 0 for p = min{n, m}. If rank A = r , then

r ≤ p. In particular if r < p, then

σ1 ≥ σ2 ≥ · · · σr > 0 = σr +1 = · · · = σp .

However, due to rounding error, the computed singular values of A

are likely to satisfy

σ1 ≥ σ2 ≥ · · · σk >  >> σk +1 ≥ · · · ≥ σp ≥ 0

for some 1 ≤ k ≤ p, where 0 <  << 1.

Numerical rank determination via SVD
Let A = UΣV ∗ be an SVD of an n × m real or complex matrix A with

Σ := diag(σ1 , σ2 , . . . σp ) ∈ Rn×m

where σ1 ≥ σ2 ≥ · · · σp ≥ 0 for p = min{n, m}. If rank A = r , then

r ≤ p. In particular if r < p, then

σ1 ≥ σ2 ≥ · · · σr > 0 = σr +1 = · · · = σp .

However, due to rounding error, the computed singular values of A

are likely to satisfy

σ1 ≥ σ2 ≥ · · · σk >  >> σk +1 ≥ · · · ≥ σp ≥ 0

for some 1 ≤ k ≤ p, where 0 <  << 1.

In such cases, we may set σj = 0, for j = k + 1, . . . , p, and state that
the numerical rank of A is k .
Numerical rank determination via SVD
Let A = UΣV ∗ be an SVD of an n × m real or complex matrix A with

Σ := diag(σ1 , σ2 , . . . σp ) ∈ Rn×m

where σ1 ≥ σ2 ≥ · · · σp ≥ 0 for p = min{n, m}. If rank A = r , then

r ≤ p. In particular if r < p, then

σ1 ≥ σ2 ≥ · · · σr > 0 = σr +1 = · · · = σp .

However, due to rounding error, the computed singular values of A

are likely to satisfy

σ1 ≥ σ2 ≥ · · · σk >  >> σk +1 ≥ · · · ≥ σp ≥ 0

for some 1 ≤ k ≤ p, where 0 <  << 1.

In such cases, we may set σj = 0, for j = k + 1, . . . , p, and state that
the numerical rank of A is k .
If the entries of A are affected only by rounding error, then we may set
 = 2 max{n, m}ukAk2 . This is the default threshold for Matlab’s rank
command which can be modified by the user.