Matrix Algebra 1 LU-decomposition 50

2. LUDECOMPOSITION

2.1 Introduction

In this chapter we use Gaussian elimination to solve the system of equations
represented by the matrix equation

Ax = b,

where A is an mxn matrix and b an mvector. This leads to the LU-
decomposition of A, which is suitable for computer computations. We are left with
a decomposition of the following form,

A = LU,

where L is a lower triangular mxm matrix and U an upper triangular mxn
matrix. This decomposition is used to solve systems of linear equations and to
find the inverse of a matrix. As we shall notice, if A is a band matrix, then the
matrices L and U are band matrices as well. This is a useful and memory-saving
property.

First we concentrate on square matrices and then we discuss the general case of
mxn matrices. The LU-decomposition of a matrix A is gotten by left multiplying it
by so called Gaussian matrices until we have an upper triangular matrix.


2.2 Constructing LU-decomposition

Gaussian matrix. Let x be an n-vector. We construct a square matrix L
k
, such
that first k elements of the vector x remain fixed and the rest are changed to
zero,



L
k
x =
x
1

x
k
0

Gaussian matrix

Matrix Algebra 1 LU-decomposition 51


L
k
= I l
k
e
k
T
, l
k
=
1
x
k
0

0
x
k+1

x
n

, x
k
0


carries out the required transformation. Note that the Gaussian matrix is
dependent on the vector x and does not generally work for other vectors.

Exercise 2.2.1 Check that L
k
x=[x
1
,...,x
k
,0,...,0]
T
.

It is easy to compute the inverse of a Gaussian matrix. The inverse is (show this)



L
k
1
=I + l
k
e
k
T
,

Multiplying the system of equations by this matrix does not change the solution.

Exercise 2.2.2 Show that the inverse of a Gaussian matrix is of this form and
that

L
k
e
i
= e
i
, if i k.

Suppose we have a matrix A that is of the following form



A = a
1
, a
2
,..., a
n

.

We transform this matrix to an upper triangular matrix by left multiplying it by
suitable Gaussian matrices. First we use a suitable matrix L
1
to transform the
first column of A to upper triangular form. Then we use a suitable L
2
to do the
same thing for the second column and so on. After n-1 steps we have an upper
triangular matrix.

1. column: We assume, that a
11
0, then choose a Gaussian matrix L
1
such
that it transforms the first row of A so that it is parallel with e
1
. Next we
left multiply the equation by the matrix



L
1
= I l
1
e
1
T
=
1 0 0
l
21
1 0
l
31
0 1 0

l
n1
0 0 1

l
i1
=
a
i1
a
11
i = 2,..., n,



l
1
T
= 0 l
21
l
31
l
n1

,
Matrix Algebra 1 LU-decomposition 52

According to exercise 1.16.1 we get



L
1
a
1
, a
2
,..., a
n

= L
1
a
1
, L
1
a
2
,..., L
1
a
n

= a
11
e
1
, L
1
a
2
,..., L
1
a
n

,

with component wise representation



L
1
A=
a
11
a
12
a
1n
0 a
22
1
( )
a
2n
1
( )
0 a
32
1
( )
a
3n
1
( )
0 a
n2
1
( )
a
nn
1
( )

where


a
ij
(1)
: t
are the new elements. The first column is now of the correct form.

2. column: If



a
22
(1)
0
we can use a Gaussian matrix that leaves the first two elements of the second
column in previous matrix intact and transforms the rest of the elements to zero.



L
2
= I l
2
e
2
T
, l
2
T
= 0 0 l
32
l
n2

,

where


l
i2
=
a
i2
1
( )
a
22
1
( )
i = 3, 4,..., n,

and the component wise representation is



L
2
=
1 0 0 0
0 1 0 0
l
32
1

0 l
n2
0 1

.
We get the product


L
2
L
1
A = L
2
a
11
e
1
, L
1
a
2
,..., L
1
a
n

= a
11
e
1
, L
2
L
1
a
2
,..., L
2
L
1
a
n

,

that is
Matrix Algebra 1 LU-decomposition 53


L
2
L
1
A=
a
11
a
12
a
13
a
1n
0 a
22
1
( )
a
23
1
( )
a
2n
1
( )
0 0 a
33
2
( )
a
3n
2
( )

0 0 a
3n
2
( )
a
nn
2
( )

.

Note that exercise 2.2.2 shows us that the first row remains the same.


k. column: We make a description of a general step after we have performed k-1
steps already. If



a
kk
(k1)
0 ,

then we can find a Gaussian matrix, that leaves the first k elements of the k
th

column intact and transforms the others to zero.






l
ik
=
a
ik
k1
( )
a
kk
k1
( )
, i = k +1,..., n


so the Gaussian matrix has the following element wise representation



! k
L
k
=
1 0 0 0
0 1 0 0
0 1 0
0 l
k+1 k

0 l
n k
0 1

1
]
1
1
1
1
1
1
1
1
k.

Finally k=n-1 and we have



L
n1
L
n2
...L
1
A = U,

where U is upper triangular as we wanted.

The product of two lower triangular matrices that have ones on the diagonal is a
lower triangular matrix that has ones on the diagonal (exercise 1.16.8), so the
matrix

L = L
n1
L
n2
...L
1
,
Matrix Algebra 1 LU-decomposition 54

is lower triangular with ones on the diagonal. Now we can write

LA =
a
11
a
12
a
1n
0 a
22
1
( )
a
2n
1
( )

0 0 0 a
nn
n1
( )

= U

where U is an upper diagonal matrix. Because the inverse of a lower triangular
matrix with ones on diagonal is a lower triangular matrix with ones on the
diagonal (exercise 1.21.5), we finally get the decomposition we were after.



A =

L
1
U = LU.

Exercise 2.2.3 The lower triangular matrix L could be easily written by using
the Gaussian matrices used to form the decomposition. Prove that



L =

L
( )
1
= I + l
i
i=1
n1

e
i
T
.

Example 2.2.1 We now find the LU-decomposition of the matrix



A =
2 2 1
2 3 1
2 1 2

, .

First we form the matrix L
1
.



L
1
=
1 0 0
1 1 0
1 0 1

and now


L
1
A =
2 2 1
0 1 0
0 1 1

Next we have L
2,




L
2
=
1 0 0
0 1 0
0 1 1

and we finally get the upper triangular matrix U and
Matrix Algebra 1 LU-decomposition 55



L
2
L
1
A =
2 2 1
0 1 0
0 0 1

= U ,

L = L
2
L
1
=
1 0 0
1 1 0
2 1 1

.
Note that the previous exercise provides us with an easy way to compute the
inverse of

L. We get

L
1
= L =
1 0 0
1 1 0
1 1 1

.

The LU-decomposition for the matrix A is



A =
1 0 0
1 1 0
1 1 1

2 2 1
0 1 0
0 0 1

= LU.

Exercise 2.2.4 Find the LU-decomposition of the matrix




A =
2 3 3
4 9 10
6 21 26

.


2.2.1. Existence of LU-decomposition

Now we want to know when the LU-decomposition of a given matrix exists. In
construction of an LU-decomposition we use the elements



a
11
, a
22
1
( )
, a
33
2
( )
,..., a
n1,n1
n2
( )
.

as denominators. These elements and

a
nn
n1
( )
are called pivot elements. According
to the construction of LU-decomposition, it should be clear that if the n-1 first
pivot elements differ from zero then the LU-decomposition of a square matrix A
exists.

The following simple condition ensures the existence of LU-decomposition.

Theorem 2.2.1. The first k leading principal minors of an nxn matrix A differ
from zero if and only if the first k pivot elements differ from zero. If the leading
principal minors of A are denoted

Matrix Algebra 1 LU-decomposition 56


A
1
= a
11
, A
2
=
a
11
a
12
a
21
a
22
A
n
= A ,
then



A
k
= a
11
a
22
1
( )
a
kk
k1
( )
, k =1, 2,.., n.

Proof. The product of matrices A and L
k
is the matrix that is received after
subtracting a scalar multiple of the k
th
row of A from some other rows of A. This
does not change the value of the determinant. The same operations are made for
every leading principal submatrix, so the corresponding principal leading minors
do not change either. We can prove the claim by computing leading principal
minors of the matrices A and

LA.

If it turns out that one of the n-1 first pivot elements has value zero, we can
exchange rows and continue the procedure.

Exercise 2.2.2. If det(A) 0, then it is possible to reorder the rows of A so that we
get nonzero pivot elements when constructing the LU-decomposition.

Proof. If on the first round of procedure we have a
11
= 0, we could choose some
other element a
k1
0. If such an elements does not exist we have



det A
( )
=
0 a
12
a
1n

0 a
n2
a
nn
= 0

which contradicts the assumption det(A) 0. Exchange the 1
st
and k
th
rows by
using the permutation matrix P
1
. Now the element a
k1
is in the upper left corner
and we are able to continue the procedure.

k
th
step: If


a
kk
k1
( )
= 0,
then we could choose some nonzero element under the pivot element,



a
ik
k1
( )
0, i > k.

If there does not exist such an element, then



det L
k1
P
k-1
L
1
P
1
A
( )
= 0 ,

and det(A) = 0 which is a contradiction. We use the permutation matrix P
k
to
exchange the i
th
and k
th
row and continue the procedure by left multiplying by
matrix L
k
.

Matrix Algebra 1 LU-decomposition 57
If the necessary exchanges of rows are done right in the beginning by using a
permutation matrix P, we have proven the existence theorem of LU-
decomposition.

Theorem 2.2.3. (LUdecomposition for nxn matrices.) Let A be a
nonsingular nxn matrix. Then there exists a permutation matrix P so that

PA = LU,

where L is a lower triangular matrix with ones on the diagonal and U an upper
triangular matrix.

2.2.4 Pivoting


The numerical efficiency of LU-decomposition can be improved by choosing the
element in every elimination cycle that has the biggest absolute value, as a pivot
element



a
kk
k1
( )


by changing the order of rows and/or columns. In partial pivoting the new pivot
element is only searched for among the elements below the original pivot element



a
kk
k1
( )
: n.

In complete pivoting the new pivot element is searched for among the elements
that have indexes ik and jk. This means that in complete pivoting we may
change the order of the variables as well as the order of the equations.


2.2.3 The solution of a system (nxn) of linear equations by using LU-
decomposition

LU-decomposition is mainly used to solve matrix equations Ax=b. First we find
the LU-decomposition of a matrix A. Partial pivoting does not change the
solution, because only the order of the equations is changed. In complete pivoting
we might change the solution by permuting it since we can change the order of
the variables x
i
. Complete pivoting leads to an equation of the form


Ax = P
T
LUx = b

that is easy to solve. The inverse of a permutation matrix P is its transpose, so
we get the equation



LUx = Pb.

We can split this into two halves.

Ly=Pb,

Matrix Algebra 1 LU-decomposition 58
Ux=y.

The first equation has a unique solution y=L
-1
Pb, because L is invertible. The
matrix U is an upper triangular matrix that has the pivot elements of A on its
diagonal. The latter equation also has a unique solution if all the pivot elements,
including the last one, are nonzero.

Previous results are summed up in the next theorem.

Theorem 2.2.4. Let A be an nxn matrix and b an nvector. If det(A) 0, the
equation Ax = b has a unique solution for every vector b.

Exercise 2.2.5 Assume that det(A) 0 and PA = LU. Show that the matrices L
and U are unique.

Example 2.2.2. Let us use the LU-decomposition to solve the equation Ax = b,
where A is the matrix in example 2.2.1 and b = [1,1,1]
T
. The LU-decomposition of
the matrix A is



A =
1 0 0
1 1 0
1 1 1

2 2 1
0 1 0
0 0 1

= LU

If we use the following notation

Ly=b and Ux=y,


we can easily solve the system of equations because the equation



Ly =
1 0 0
1 1 0
1 1 1

y
1
y
2
y
3

=
1
1
1

,

is simple when we start from the first equation. The solution is

y= [1,0,0]
T
.

Then we solve the equation Ux=y. This is not hard either, because U is an upper
triangular matrix.



2 2 1
0 1 0
0 0 1

x
1
x
2
x
3

=
1
0
0

x
3
= 0, x
2
= 0, x
1
= 1 / 2.
The solution is

Matrix Algebra 1 LU-decomposition 59


x =
1 2
0
0

.

Theorem 2.2.5 Let A and B be square matrices. Now det(AB)=det(A)det(B).

Proof. Let

det(A) 0 . We assume that A = P
T
LU is the LU-decomposition of
square matrix A. The determinant of the matrix AB can be written in the form

det(AB)=det(P
T
LUB) = (-1)
s(A)
det(LUB)

where s(A) is the amount of rows exchanged by multiplying by a permutation
matrix P. By using exercises 1.24.6-9 we can form the following chain of
equations

det(AB) = (-1)
s(A)
det(LUB) = (-1)
s(A)
det(UB)=(-1)
s(A)
det(U)det(B)

= (-1)
s(A)
det(LU)det(B) = det(P
T
LU)det(B) = det(A) det(B).

If

det(A) = 0 , then, as will be shown in chapter 2.5, we can construct for the
matrix A a corresponding LU-decomposition, where the last row of matrix U
consists of zeros. In this case we have

det(AB) = (-1)
s(A)
det(U) det(B)=0,

because det(U)=0.


Exercise 2.2.6 Construct the LU-decomposition of the matrices



a) A =
2 2 2
2 3 1
4 2 2

and


b) A =
0 0 1
1 0 1
1 3 1

and solve the equation Ax = [1 2 3]
T
.




2.3. Finding the inverse of a matrix by using LU-
decomposition

Matrix Algebra 1 LU-decomposition 60
LU-decomposition gives us new information about the inverse of a matrix: if
det(A) 0, then A has an inverse that can be computed by solving the system of
equations,



Ax
i
= e
i
i = 1, 2,..., n

This could also be presented in the form



A x
1
, x
2
,..., x
n

= e
1
, e
2
,..., e
n

= I,

so AX = I. The properties of the determinant now say that

det(AX) = det(A) det(X) = 1 = det(I),

so det(X) 0, and as seen, theorem 2.2.4 could be used to find an nxnmatrix Y
such that it satisfies the equation

XY = I.

We left multiply the equation by A:



AX
I

Y = A Y = A

So AX = XA = I, and that is X = A
1
.

We have almost proved the following theorem:

Theorem 2.5.1 Let A be an nxnmatrix. The following conditions are equivalent

(a) det(A) 0
(b) there exists a matrix A
-1

(c) the equation Ax = b has a unique solution
x = A
-1
b, b
(d) Ax = 0 x = 0

Proof.
(a) (c) (b) Follows from the theorem 2.2.4 and what is said above.
(b) (a): 1= det(I) = det(AA
1
) = det(A)det(A
1
) det(A) 0.
(c) (d) : trivial (choose b=0).
(d) (a) : Suppose that antithesis is valid, that is: (d) holds and
det(A) = 0.

We construct the LU-decomposition of A as earlier. Because det(A) = 0, there is a
pivot element with value zero. Let the first pivot element that is zero be



a
kk
(k1)
= 0

The elements underneath it are also zero. We have the matrix
Matrix Algebra 1 LU-decomposition 61



k1 1 n k

LA =
A
1
a B
0 0 C

k1
n k+1


The submatrix A
1
is upper triangular and its diagonal elements are



a
11
, a
22
1
( )
,..., a
k1,k1
k2
( )
0 .

This means that A
1
-1
exists. We choose



x =
A
1
1
a
1
0

} k1
} 1
} nk
0.

Left multiplying x by

LA gives us:



A
1
a B
0 0 C

A
1
1
a
1
0

=
a+ a
0

=
0
0

= 0.

which contradicts (d), because

Ax = 0 follows from equation

LAx = 0 . So
(d) (a) and the proof of the theorem is completed.


2.4 Computation times of LUdecompositions

Constructing the LUdecomposition of a nonsingular nxn matrix takes (n
3
)/3
flops. Solving the equation Ax = LUx = b requires the solutions of the equations



Ly = b y = L
1
b

and

Ux = y

Each of which require (n
2
)/2 flops. All together this takes (n
3
)/3 + n
2
flops.

Because det(A)=det(U)= +u
11
u
22
...u
nn
, the determinant could be computed in
(n
3
)/3 flops, if only the highest order of n is considered.

If the equation Ax = b is solved for different b, the construction of the LU
decomposition is only carried out once. If we solve the equation with p different
vectors, we need
Matrix Algebra 1 LU-decomposition 62



n
3
3
+ p n
2


flops to get all the answers. Computing the inverse A
1
demands that we solve
for the n vectors e
i
, i=1,..,n, (chapter 2.6) so it takes



n
3
3
+ n n
2
=
4
3
n
3


flops.

To save memory space we could replace the matrix A with its LU-decomposition
in the following way:



a
11
a
1n
a
21
a
2n

a
n1
a
nn

u
11
u
1n
l
21
u
22
u
2n

l
n1
l
n2
u
nn

.

The memory space is sufficient because the diagonal elements of L are ones and
there is no need to use memory for them.


2.5 LUdecomposition of singular and mxn matrices




A =
2 1 2 1
2 1 3 2
2 1 3 2
2 1 3 2
2
1
1
1

.
Now


L
1
=
1 0 0 0
1 1 0 0
1 0 1 0
1 0 0 1

.

Let A be an mxn matrix. We construct the LU-decomposition as we did for square
matrices by using Gaussian matrices. We transform the columns one by one to
the required form and use partial pivoting when necessary. The difference now is
that there might be a situation in which the pivot element and all the elements
under it are zero. In this kind of situation we continue the elimination as we did
so far, but in the same row and the next column. The next example should clarify
the procedure.

Example 2.5.1 Consider the matrix
Matrix Algebra 1 LU-decomposition 63
1
st
step



L
1
A =
2 1 2 1
0 0 1 1
0 0 1 1
0 0 1 1
2
1
1
1

we see, that


a
22
(1)
= 0 ja a
i2
(1)
= 0, kun i = 3, 4.

We continue the procedure in the third column and transform the elements
under the second element in this column to zero. Now we get



L
2
=
1 0 0 0
0 1 0 0
0 1 1 0
0 1 0 1

and


L
2
L
1
A =
2 1 2 1
0 0 1 1
0 0 0 0
0 0 0 0
2
1
0
0

= U

Generally the solution is of the form



0 0 0 * x x x x x x x
0 0 0 0 0 * x x x x x
0 0 0 0 0 0 0 0 * x x

0 0 0 0 0 0 0 0 * x x
0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0

1
]
1
1
1
1
1
1
1
1
1
1
1

m r


Where the elements denoted with * are nonzero and elements denoted with x
could be zero or something else.

We end up with the equation,

PA = LU,

where P is an mxm permutation matrix, L an mxm nonsingular lower triangular
matrix that has ones on the main diagonal and U is an mxn upper triangular
matrix of the form represented above. It could be shown that the number of
Matrix Algebra 1 LU-decomposition 64
nonzero rows is unique and does not depend on the method used to get the form
above.


2.6 Application: Solving partial differential equations by the
difference method


Difference method. We consider a simple boundary value problem. We want to
find the function v(x) with v(0)=v(1)=0 such that

v''(x) = f(x)

on the interval [0,1] when f is a given function.

The problem can be solved in the following way by using the difference method.
We divide the interval [0,1] into n+1 subintervals

x
0
=0, x
1
=h, x
2
=2h, ..., x
i
=ih, ..., x
n+1
=1,

where the length of every subinterval is h=1/(n+1). The values of the function at
the points of the partition are denoted:



v(x
i
) = v
1
, f (x
i
) = f
i
, i = 0,1, ..n+1.

The boundary conditions say that v
0
= v
n+1
= 0. Taking difference quotients
gives us some approximations for the derivatives



v'(x
i
) (v
i+1
v
i1
) / 2h, v''(x
i
) (v
i+1
2v
i
+ v
i1
) / h
2
.


At each point we form the equation



v''(x
i
) = (v
i+1
2v
i
+ v
i1
) / h
2
= f
i
i = 1, 2, ..., n.

Write down this equation in matrix notation which has v
i
, i=1,,n, as variables.
You will get a band matrix. What is the band width of this matrix? Is it
symmetric and positive definite? The bigger the number n is the more accurate
the solution is. Solve the equation when n=5 and a) f(x) = 1, b) f(x)=x(1-x).