Prepared by:

Engr. Romano A. Gabrillo

MEngg - MfgE
Basically there are two types of methods namely
direct methods and indirect methods available
for the solution of linear system of equations.

In this chapter the following methods and
implementation procedures of these methods on
computer are to be discussed:
Direct Methods:
Gaussian Elimination Method
Gauss-Jordan Method
Indirect or Iterative Method
Jacobi Method
Gauss-Seidel Method
Consider a linear system of equations as shown
below:
a
11
x
1
+ a
12
x
3
+ a
12
x
3
+ . . . a
1n
x
n
= b
1

a
21
x
1
+ a
22
x
2
+ a
23
x
3
+ . . . a
2n
x
n
= b
2
. . . . . . .
. . . . . . .
a
n1
x
1
+ a
n2
x
2
+ a
n3
x
3
+ . . . a
nn
x
n
= b
n


Thus,
[A] {x} = {b}
n x n n x 1 n x 1
Hence, the matrix multiplication is compatible
with the solution of linear algebraic equations. If
{b} =! {0} then the system of equation given in
previous eqn. is known as non-homogeneous
system of equations. Homogeneous system of
equations lead to eigenvalue problems.

Herein we consider only non-homogeneous
equations. Basically there are two types of
methods available viz. Direct method and
Indirect methods or iterative methods.
1. Gaussian Elimination
2. Gauss-Jordan Method

Indirect Methods
1. Jacobi Method
2. Gauss Seidel Method
Gaussian elimination is the first method
usually presented in algebra for the solution
of simultaneous linear equations.

The main aim of this method is to reduce a
set of n equations in n unknowns to an
equivalent triangular set (an equivalent set is
a set having identical solution values) which
is then easily solved by back substitution.
This method, hence, consists of two steps.
1. Triangularization
2. Backward Substitution

Consider the equations in first slide. They may be
written as:
E
1
0
= a
11
x
1
+ a
12
x
2
+ + a
1n
x
n
= b
1
E
2
0
= a
21
x
1
+ a
22
x
2
+ + a
2n
x
n
= b
2
. . . . .
. . . . .
E
i
0
= a
i1
x
1
+ a
i2
x
2
+ + a
ij
x
j
+ . . . + a
in
x
n
= b
i
. . . . .
. . . . .
E
n
0
= a
n1
x
1
+ a
n2
x
2
+ + a
nn
x
nn
= b
n

The first equation E
1
0
of the previous slide is
divided by the coefficient of x
1
in that
equation to obtain
E
1
1
= E
1
0
/a
11


x
1
+ a
12
x
2
+ a
13
x
3
+ a
1n
x
n
= b
1

a
11
a
11
a
11
a
11



Then, multiply it by the coefficient of x
1
in
E
2
0
and resulting equation is subtracted from
E
2
0
previously thus eliminating x
1
from E
2
0

1 1
3
1
3 2
1
2
1
1 1
0 1
1 1
3 3 2
1
2
1
1 1
0 1
1
1
3
1
3 2
1
22
11
1 21
2
11
1 21
2 3
11
13 21
23 2
11
12 21
22
1
1 21
0
2
1
2
...
...
,
...
...
n n nn n n
n n n
n n nn i i
i i i
i n in i
n
n
n
b x a x a x a
E a E E
on so and
b x a x a x a
E a E E
Similarly
b x a x a x a or
a
b a
b
x
a
a a
a x
a
a a
a x
a
a a
a
E a E E
= + + + =
=
= + + + =
=
= + + +
=
|
|
.
|

\
|
+ +
|
|
.
|

\
|
+
|
|
.
|

\
|
=
=
1 1
2
1
1 1 1
2
1
1
2
1
2
1
2
1
22
1
2
1 1 1 12 1 1
... ... 0
... ... 0
. . . . .
... ... 0
' ' ... ' ... ' '
n n nn j nj n n
i n in j ij i i
n n j j
n n j j
b x a x a a E
b x a x a a E
b x a x a a E
b x a x a a x E
= + + +
= + + +
= + + +
= + + +
The equation used to eliminate the unknowns
in the equation which follow it is called the
pivot equation.

In the pivot equation the coefficient of the
unknown which is to be eliminated from
subsequent equation is known as the pivot
coefficient (a
11
in the preceding steps).

Following the above steps considering second
equation from above as pivot-equation and
repeating the same steps to eliminate x
2
from
all the equations following this pivot equation
we, get:
n
n
nn n
n n n n
n
n
n
n n
n n
b x a x a E a E E
b x a x a E a E E
b x a x a x a E E
b x a x a x E E
2 2
3
3
2
2
2
2
2
3
3
3
33
2
2
2 22
2
3
2
2 2 3
23
2
2 22 2
2
2
1 1 2 12 1
1
1
2
1
... 0 0 ' '
. . . . .
... 0 0 ' '
... 0 ' / '
' ' ... '
= + + + = =
= + + + = =
= + + + + = =
= + + + =
n
n
n
nn
n
n
n
n
n
n n
b x a
b x a x a x
b x a x a x
b x a x a x a x
1 1
3
3
3
3
4
34
3
3
2
2
2
2
3
23
2
2
1 1 3 13 2 12 1
... ... ... 0 0
. . . . .
... 0 0
... 0
' ' ... ' '

= + +
= + + + +
= + + +
= + + + +
(
(
(
(
(
(

=
(
(
(
(
(
(

=
(
(
(
(
(
(

n
n
n
nn
n
n
n
b
b
b
x
x
x
a
a a
a a a
1
2
2
1
2
1
1
2
2
23
2
1 13 12
...
...
'
...
...
0 ... ... 0 0 0
...
... ... ... 0
1 0
' ... ... ... ' ' 1
After the triangular set of equations has been
obtained, the last equation in this equivalent set
yields the value x
n
is directly as:
x
n
= b
n
n-1
/ a
n(n-1)
n-1


This value is then substituted into the next-to-
last equation of the triangular set to obtain a
value of x
n-1
as:
x
n-1
= b
n-1
n-1
- a
n(n-1)
n-1
x
n


And so on. This is the back substitution
procedure referred to earlier.
Since there are four equations number of
iterations required for triangularization is 3.

(
(
(
(

21
26
32
24
4
3
2
1
9 4 4 0
4 5 . 6 5 2
4 5 10 4
0 2 4 8
0
4
0
3
0
2
0
1
x
x
x
x
E
E
E
E

(
(
(
(

=
=
=
=
21
20
20
3
4
3
2
1
9 4 4 0
4 6 4 0
4 4 8 0
0 25 . 0 5 . 0 1
) ( 2
) ( 4
8 /
0
4
1
4
1
1
0
3
1
3
1
1
0
2
1
2
0
1
1
1
x
x
x
x
E E
E E E
E E E
E E
(since a
41
= 0)

(
(
(
(

=
=
=
=
11
10
5 . 2
3
4
3
2
1
7 2 0 0
2 4 0 0
5 . 0 5 . 0 1 0
0 25 . 0 5 . 0 1
4
) ( 4
8 /
2
2
1
4
2
4
2
2
1
3
2
3
2
2
2
2
1
1
2
1
x
x
x
x
E E E
E E E
E E
E E

(
(
(
(

=
=
=
=
6
5 . 2
5 . 2
3
4
3
2
1
6 0 0 0
5 . 0 1 0 0
5 . 0 5 . 0 1 0
0 25 . 0 5 . 0 1
2
4 /
3
3
2
4
3
4
2
3
3
3
2
2
3
2
2
1
3
1
x
x
x
x
E E E
E E
E E
E E
2 ; 3 0 25 . 0 5 . 0
1 ; 5 . 2 5 . 0 5 . 0
2 ; 5 . 2 4 5 . 0
1 ; 6 6
4 3 2 1
3
1
3 4 3 2
3
2
3 3
3
3
4 4
3
4
= = + + +
= = + +
= = +
= =
x x x x E
x x x x E
x x x E
x x E
Reviewing the procedure outlined in
triangularization it can be seen that the
elements of the reduced matrix A can be
reduced matrix A can be written directly
from the original matrix A, using the formula
as:
a
ij
= a
ij
a
ik
(a
kj
) k<=j<=m
a
kk
k+1<=i<=n
After obtaining the augmented matrix of the
equivalent triangular set of equations, the x
i

values are obtained by back substitution.

In order to program this in a computer we
can write:

) (
1
1
1
1

=
k
ik
k
kk
k
kj
k
ij
k
ij
a
a
a
a a
1
1 1
1

=
k
kk
k
k
k
ik
k
i
k
i
a
b a
b b
i = row number of matrix
j = column number of matrix
k = number identifying pivot row
n = number of rows in a matrix
m = number of columns in a matrix

If a
ik
k-1
is equal to zero, this step can be
skipped. The back substitution procedure
may be generalized in the form of the
following set of equations as
x
j
values are stored in the same place of bi.
Let us see the same example before and see
how the Gaussian elimination is performed in
a computer.
1 ... , 2 , 1
/
1
=
=
=

+ =
n n i
x a b x
a b x
n
i j
j ij i i
nn n n

21
26
32
24
9 4 4 0
4 5 . 6 5 2
4 5 10 4
0 2 4 8
21
20
20
3
9 4 4 0
4 6 4 2
4 4 8 4
0 25 . 0 5 . 0 8
11
10
5 . 2
3
7 2 4 0
2 4 4 2
5 . 0 5 . 0 8 4
0 25 . 0 5 . 0 8
6
5 . 2
5 . 2
3
6 2 4 0
5 . 0 4 4 2
5 . 0 5 . 0 8 4
0 25 . 0 5 . 0 8
In a triangularization process, we perform the
matrix A into [L] lower triangular matrix and
[U] upper triangular matrix consisting of 1 on
the leading diagonal as shown in the final
resulting matrix during the triangularization
process:
(
(
(
(

1 6 2 4 0
5 . 0 1 4 4 2
5 . 0 5 . 0 1 8 4
0 25 . 0 5 . 0 1 8
[L] =





[U] =
(
(
(
(

6 2 4 0
4 4 2
8 4
8
(
(
(
(

1
5 . 0 1
5 . 0 5 . 0 1
0 25 . 0 5 . 0 1
i.e [A] = [L][U]
and [L][U]{X} = {b}
or [U]{X} = [L]
-1
{b}

To find the determinant of (A)
[A] = [L][U]
det (A) = det (L) det [U]
and det [L] = product of the leading terms
=
= 8 x 8 x 4 x 6 = 1536
ii
n
i
L
1
[
where L
ii
are the same as leading diagonal terms of
the modified matrix of [A] during the
triangularization process.

Hence to find the determinant of the matrix
[A] after the triangularization step multiply all
the diagonal terms as:

ii
n
i
a A
1
) det(

[ =
An electrical network is defined by the following
control systems equations for the three currents.
These equations are shown below:

i
1
(t) + 2i
2
(t) + 5i
3
(t) =4
5i
1
(t) + 1i
2
(t) + 3i
3
(t) = 0
5
7i
1
(t) + 3i
2
(t) + 1i
3
(t) =3
2
Solve the values of the currents using the method
of Gaussian Elimination.
The main objective of this method is to diagonalize
the given matrix. This eliminates the necessity of
using back substitution process employed by Gausss
Method. This is suitable for solving as many as 15 to
20 simultaneous equations with seven to ten
significant digits used in the arithmetic operations of
the computer.

In Gaussian elimination the unknown is eliminated
from all equations except the pivot equation whereas
in Gauss-Jordan method the unknown is eliminated
from all equations except the pivot equation whereas
in Gauss-Jordan method the unknown is eliminated
from all the equations.
To illustrate the method let us take the same
example before.

(
(
(
(

21
26
32
24
4
3
2
1
9 4 4 0
4 5 . 6 5 2
4 5 10 4
0 2 4 8
0
4
0
3
0
2
0
1
x
x
x
x
E
E
E
E

(
(
(
(

=
=
=
=
21
20
20
3
4
3
2
1
9 4 4 0
4 6 4 0
4 4 8 0
0 25 . 0 5 . 0 1
) ( 2
) ( 4
8 /
0
4
1
4
1
1
0
3
1
3
1
1
0
2
1
2
0
1
1
1
x
x
x
x
E E
E E E
E E E
E E

(
(
(
(

=
=
=
=
11
10
5 . 2
75 . 1
4
3
2
1
7 2 0 0
2 4 0 0
5 . 0 5 . 0 1 0
25 . 0 0 0 1
4
) ( 4
8 /
5 . 0
2
2
1
4
2
4
2
2
1
3
2
3
1
2
2
2
2
2
1
1
2
1
x
x
x
x
E E E
E E E
E E
E E E

(
(
(
(

=
=
=
=
6
5 . 2
25 . 1
75 . 1
4
3
2
1
6 0 0 0
5 . 0 1 0 0
25 . 0 0 1 0
25 . 0 0 0 1
2
4 /
5 . 0
0
3
3
2
4
3
4
2
3
3
3
3
3
2
2
3
2
3
3
2
1
3
1
x
x
x
x
E E E
E E
E E E
E E E

(
(
(
(

=
=
=
=
1
2
1
2
4
3
2
1
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
6 /
5 . 0
25 . 0
25 . 0
3
4
4
4
4
4
3
3
4
3
4
4
3
2
4
2
4
4
3
1
4
1
x
x
x
x
E E
E E E
E E E
E E E






For improved accuracy similar to Gaussian
elimination method one can use partial and
full pivoting.

=
1
2
1
2
} {x
Matrix Inverse by Gauss-Jordan Method
If [A] is nonsingular
[A][X] = [I]
then [X] = [A]
-1

Solve the equation by Gauss-Jordan Method
by transforming [A] to [I]
The modified right hand side vector will give
the inverse of [A].
After first elimination
(
(
(

1 0 0
0 1 0
0 0 1
1 0 2
1 1 4
1 2 3
(
(
(

1 0 3 / 2
0 1 3 / 4
0 0 3 / 1
3 / 5 3 / 4 0
3 / 7 3 / 5 0
3 / 1 3 / 2 1
After third elimination
(
(
(

1 5 / 4 5 / 2
0 5 / 3 5 / 4
0 5 / 2 5 / 1
5 / 1 0 0
5 / 7 1 0
5 / 3 0 1
(
(
(

5 4 2
7 5 2
3 2 1
1 0 0
0 1 0
0 0 1
(
(
(

5 4 2
7 5 2
3 2 1
] [
1
A
An electrical network is defined by the following
control systems equations for the three currents.
These equations are shown below:

i
1
(t) + 2i
2
(t) + 5i
3
(t) =4
5i
1
(t) + 1i
2
(t) + 3i
3
(t) = 0
5
7i
1
(t) + 3i
2
(t) + 1i
3
(t) =3
2
Solve the values of the currents using the Gauss-
Jordan Method.
When the matrix is sparse, the iterative
methods may prove be more efficient than
the direct methods. This section reviews the
two iterative techniques and their computer
implementation.

Indirect Methods:
Jacobi Method
Gauss-Seidel Method

In order to apply this iterative technique, the
equation must satisfy diagonal dominance
criteria:

= =
>
n
i j j
ij j
a ai
, 1

|
|
|
|
|
.
|

\
|
n n nn n n
n
n
b
b
b
x
x
x
a a a
a a a
a a a
... ...
... ...
... ... ... ... ...
... ...
... ...
2
1
2
1
2 1
2 22 21
1 12 11
nn
n
nn
n
nn
n
nn
n
n n
n n
a
b
a
x
a
x a
a
x a
a
b
a
x a
a
x a
x
a
x a
a
b
a
x a
a
x a
a
x a
x
= + +
= + +
= + +
...
... ... ... .... ...
...
...
2 2 1 1
22
2
22
2
22
3 23
2
22
1 21
11
1
11
1
11
3 13
11
2 12
1
(

=
=
=
=

= =
= =
= =
= =
n
i j j
j ij i
ij
i
n
i j j
ii
j
j
i
ii
i
i
n
j j
j
j
n
j j
j
j
i
x a b
a
x or
a
x a
a
b
x
a
x a
a
b
x
a
x a
a
b
x
, 1
, 1
2 , 1
22
2
22
2
2
1 , 2
22 11
1
1
1
Guess initially all the unknowns x
1
(0)
and
solve each equation in turn for x
i
1
using x
i
0
a:




or the values of x
1
(r)
at rth iteration can be
found using the values of x
i
(r-1)
of (r-1)th
iteration.


|
|
.
|

\
|
=

= =
n
i j j
j ij i
ii
i
x a b
a
x
, 1
) 0 ( ) 1 (
1
Relative error:



If one needs p decimal places accuracy


i
r
i
r
i
x all for x x
1
1
| | c <

2
1
c <

r
i
r
i
r
i
x
x x
) 2 (
2
) 1 (
1
10 5
10 5
+
+
=
=
p
p
x
x
c
c
Solve the following using Jacobi Method:




The matrix is strictly diagonally dominant, i.e.

(
(
(

12
10
16
5 2 1
1 3 1
2 1 4
3
2
1
x
x
x
| |
| |
| | 3 5
2 3
3 4
32 31 33
23 21 22
13 12 11
= + > =
= + > =
= + > =
a a a
a a a
a a a
{ }
{ }
{ }
) 1 (
2
) 1 (
1
) (
3
) 1 (
3
) 1 (
1
) (
2
) 1 (
3
) 1 (
2
) (
1
2 12
5
1
10
3
1
2 16
4
1



=
=
=
k k k
k k k
k k k
x x x
x x x
x x x
4 . 2 5 / 12
333 . 3 3 / 10
4 4 / 16
0 ; 0 ; 0
) 1 (
3
) 1 (
2
) 1 (
1
) 0 (
3
) 0 (
2
) 0 (
1
= =
= =
= =
= = =
x
x
x
x x x
X\K 0 1 2 3 4 5 6
x
1
0 4.0 1.9667 3.5667 2.5894 3.2656 2.8227
x
2
0 3.333 1.20 2.5889 1.6355 2.2532 1.8355
x
3
0 2.4 2.4 3.2667 0.6511 1.2279 0.8456
X\K 7 8 9 10 11
x
1
3.1183 2.9217 3.0523 2.9654 3.0231
x
2
2.1106 1.9268 2.0487 1.9676 2.0215
x
3
1.1013 0.9321 1.0449 0.9701 1.0199
Apparently the scheme converges slowly to 3,
2, 1.

From this iterative technique it is clear that
this method converges to the correct answer
but the convergence rate is slow.
Start
Read Number of
Equations Max
Iterations and
Tolerance
Read (A) Matrix,
Right Hand Side
Vector {b} &
Starting Vector
{x}
0

Modify [A] Matrix as
[A] = 0 a
12
.... a
1n

a
11
a
11

a
21
0 .... a
2n

a
22
a
22

a
n1
...... 0
a
nn


{b} = b
1
/ a
11

.
.

.
b
n/
a
nn

A
A

= =

=
=
1
, 1
1
300
k
i
k
i
n
i j j
k
j ij i
k
i
x x dif
x a b x
iterations do
If dif
< Tol
Print
Result
End
An electrical network is defined by the following
control systems equations for the three currents.
These equations are shown below:

i
1
(t) + 2i
2
(t) + 5i
3
(t) =4
5i
1
(t) + 1i
2
(t) + 3i
3
(t) = 0
5
7i
1
(t) + 3i
2
(t) + 1i
3
(t) =3
2
Solve the values of the currents using Jacobi
Method.
A slight improvement of the Jacobi Method is
Gauss-Seidel iteration (also called method of
successive displacement). In this method the
updated values of x
i
are used instead of the
values of the previous iteration.

For example:
) (
1
) 0 (
2
1
11
) 1 (
1 j
n
j
ij
x a b
a
x

=
=
After this equation x
1
(1)
is known and hence
to find x
2
(1)
, x
1
(1)
(updated value) is used
instead of the value of the previous iteration
as:


or



) ... (
1
) 0 (
2
) 0 (
4 24
) 0 (
3 23
) 1 (
) 1 ( 21 2
22
) 1 (
2 n n
x a x a x a x a b
a
x =
) (
1
) 0 (
3
2
) 1 (
1
1
2 2
22
) 1 (
2 k
n
k
k k
k
k
x a x a b
a
x

= =
=
The equation above may be written in an
iterative form as:
) (
1
) 0 (
1
) 1 (
1
1
) 1 (
2 k
n
i k
ik k
i
k
ik i
ii
x a x a b
a
x

+ =

=
=
) (
1
) 1 (
1
) (
1
1
) (
+ =

=

=
r
j
n
i j
ij
r
j
i
j
ij i
ii
r
i
x a x a b
a
x
In fact for those cases when Jacobi Method
does converge, Gauss-Seidel iteration can
usually be expected to converge even faster.
If Gauss-Seidel Method converges it usually
does so only linearly.

Even in this case the equations are written in
such a way that the matrix is diagonally
dominant.
The equations can be rearranged such that
the matrix is diagonally dominant as:
16 2 4
12 5 2
10 3
3 2 1
3 2 1
3 2 1
= + +
= + +
= + +
x x x
x x x
x x x

(
(
(

12
10
16
5 2 1
1 3 1
2 1 4
3
2
1
x
x
x
The iterations are shown in the table below:
| |
| |
| | ) 2 12 ) 5 / 1 (
) 10 ) 3 / 1 (
) 2 16 ) 4 / 1 (
) (
2
) (
1
) (
3
) 1 (
3
) (
1
) (
2
) 1 (
3
) 1 (
2
) (
1
r r r
r r r
r r r
x x x
x x x
x x x
=
=
=

x\r 0 1 2 3 4 5
x
1
0 4 3.1 3.0083 3.0002 3.0
x
2
0 2 2.0333 2.012 2.002 2.0
x
3
0 0.8 0.9667 0.9935 0.99916 1.0
It is seen that convergence is much faster
than Jacobi.
c <

) (
) 1 ( ) (
r
i
r
i
r
i
x
x x
An electrical network is defined by the following
control systems equations for the three currents.
These equations are shown below:

i
1
(t) + 2i
2
(t) + 5i
3
(t) =4
5i
1
(t) + 1i
2
(t) + 3i
3
(t) = 0
5
7i
1
(t) + 3i
2
(t) + 1i
3
(t) =3
2
Solve the values of the currents using the Gauss-
Seidel Method.
Schedule of Mid Exam Next Week

Coverage:
Chapter 3 Only