Computational Methods for

Engineers
02 Linear Algebraic Equations
Fall 2018
 Systems of Linear Algebraic Equations
Solve the simultaneous equations Ax = b
• It is almost impossible to carry out numerical analysis of any sort
without encountering simultaneous equations
• Systems of linear equations arise for example in
 Electrical network problems, structures, heat flow, electromagnetic fields, ..
 Fitting approximating functions, and
 Systems of finite difference equations that arise in the numerical solution of
differential equations: finite difference, finite element, ..
• Equation sets arising from physical problems are often very large,
consuming a lot of computational resources (e.g., storage and/or
computation time)
NGN 509 2
 Systems of Linear Algebraic Equations
• Solve the simultaneous equations Ax = b
• Solution of n linear, algebraic equations in n unknowns
A11x1 + A12x2 + ··· + A1nxn = b1
A21x1 + A22x2 + ··· + A2nxn = b2
A31x1 + A32x2 + ··· + A3nxn = b3
...
An1x1 + An2x2 +· · ·+ Annxn = bn
where the coefficients Ai j and the constants bj are known, and xi represent the unknowns
• In matrix notation the equations are written as

Ax = b Augmented coefficient matrix

NGN 509 3
Matrix & Vector
• (n x m) matrix

• Column vector (n x 1) and raw vector (1 x n)

• Transpose of A(n x m), A’ or 𝐴𝐴𝑇𝑇 , becomes (m x n)

• Review of matrix and determinant – property and operation
• Linear Algebra (sec 1.2, pp22-30): Read: Matrix & Determinants
NGN 509 4
Matrix & Vector
• (n x n) square matrix, diagonal matrix, identity matrix

• Triangular square matrix: upper & lower triangular matrix

NGN 509 5
Matrix Operation
• A+B = B+A
• (A+B)+C = A+(B+C)
• (AB)C = A(BC)
• (A+B)C = AC+BC
• AB ≠ BA
• Matrix division is not a defined operation; but
• Inverse of A, 𝐴𝐴−1 , if A is square and nonsingular (i.e.; determinant of
A, |A| ≠ 0)
• AA−1 = A−1 A = I, where I is Identity Matrix

NGN 509 6
 Linear Algebraic Equations - Uniqueness
• A system of n linear equations in n unknowns has a unique solution,
provided that the determinant of the coefficient matrix is nonsingular,
i.e., if |A| ≠ 0.
• The rows and columns of a nonsingular matrix are linearly
independent: no row (or column) is a linear combination of other
rows (or columns)
• If the coefficient matrix is singular, the equations may have an infinite
number of solutions, or no solutions at all, depending on the constant
vector
 2x + y = 3 and 4x + 2y = 6
 2x + y = 3 and 4x + 2y = 0

NGN 509 7
 Systems of Linear Algebraic Equations
• System of linear equations can be expressed very compactly in terms
of matrix

• Two approaches to solve the equations

 Direct Methods: Based on algebraic elimination; e.g., Gauss elimination
 Iterative Methods: assume a trial solution, and iterate to convergence; e.g.,
succession-over-relaxation (SOR)
NGN 509 8
Direct Methods for LAE
• Cramer’s rule (1.3.1, p31-32) – direct method for solving system of
linear equations, Ax=b
• Requires numerous determinant evaluation
• Highly inefficient
• Number of multiplications and divisions required is N=(n-1) (n+1)!
• For small system of 10 equations, n=10, N=360,000,000
• For system of 100 equations, n=100, N=10157
• Elimination methods
• (Gauss) elimination
• Gauss-Jordan elimination
• LU decomposition (LU factorization)
• Typical elimination method requires N=𝑛𝑛3 + 𝑛𝑛2 − 𝑛𝑛; e.g., n=10, N=1090, and
n=100, N=1,009,900, which is much more manageable
NGN 509 9
 Direct (Elimination) Methods
• Gauss elimination is the most familiar method for solving simultaneous
equations. It consists of two parts:
• the forward elimination phase, and
• the Back Substitution of solution phase
• Simple Elimination method (Guass Elimination, p32-36, 1.3.2) – row
operation
 Any row (equation) may be multiplied by a constant (scaling)
 The order of the rows (equations) may be interchanged (pivoting)
 Any row (equation) can be replaced by a weighted linear combination of that row
(equation) with any other row (equation) (elimination)

• Details of Direct (Elimination) Methods:

NGN 509 10
Gauss Elimination Methods
• The elimination procedure including scaled pivoting, is commonly
called Gauss elimination
• It is the most important and most useful direct elimination method
for solving systems of linear algebraic equations
• Pivoting is an essential element of Gauss elimination to avoid zero
pivot elements is always required
• Scaled pivoting to decrease round-off errors, while very desirable in
general, can be omitted at some risk to the accuracy of the solution
• Example 1.10 illustrates the complete Gauss elimination algorithm
• Gauss-Jordan method, matrix inverse method, LU factorization
method, Thomas algorithm, etc, are all modifications or extensions of
the Gauss elimination method
NGN 509 11
 LU Factorization (Decomposition)
• Matrices (like scalars) can be factored into the product of two other
matrices in an infinite number of ways

• diagonal elements of either L or U makes the factoring unique.

NGN 509 12
LU Decomposition (Factorization)
• Decompose the original matrix to two
• L: lower matrix and
• U: upper matrix
• Using A=LU to Ax=b, gives LUx=b
• multiple both sides by L−1
• L−1 LUx = IUx = Ux = L−1 b
• Or Ux = b’ (where b′ = L−1 b)
• Multiple both sides of b′ = L−1 b by L
• Lb’ = LL−1 b  Lb’ = Ib =b  Lb’ = b
• Forward substitution on Lb’=b for b’; (L and b’ are known)
• Then back substitution on Ux=b’ for x (Using b’ and U)
• Solve the problem repeatedly with the same matrix A and different RHS b
vector (only do LU-decom once) NGN 509 13
Example – LU Decomposition

NGN 509 14
LU Decomposition – for L and U
• From the matrix A to L and U (4x4 as example)

• Doolittle’s method*
• Choleski’s method

NGN 509 15
LU Decomposition – for L and U: Doolittle’s method
• From the matrix A to L and U (3x3 as example)

• A=LU

NGN 509 16
LU Decomposition – for L and U: Doolittle’s method

• Apply Gauss elimination:

NGN 509 17
LU Decomposition – for L and U: Doolittle’s method

• Continue Gauss elimination:

NGN 509 18
LU Decomposition – for L and U: Doolittle’s method
• The matrix U is identical to the upper triangular matrix that results
from Gauss elimination.
• The off-diagonal elements of L are the pivot equation multipliers
used during Gauss elimination; i.e., Li j is the multiplier that
eliminated Ai j

or

NGN 509 19
LU Decomposition – example using matlab
• Problem

NGN 509 20
 Pitfall of Elimination Methods – Roundoff Error
• All well-posed nonsingular numerical problems have an exact solution.
• In theory, the exact solution can always be obtained using fractions or
infinite precision numbers (i.e., infinite number of significant digits).
• However, all practical calculations are done with finite precision
numbers which necessarily contain round-off errors.
• The presence of round-off errors alters the solution of the problem.
• Round-off errors can never be completely eliminated; but can be
minimized by using high precision arithmetic and pivoting (see 2nd half
of ex1.18)

NGN 509 21
 Pitfall of Elimination Methods – Roundoff Error
• Round-off errors occur when (exact) infinite precision numbers
are approximated by finite precision numbers
• In most computers, single precision contains about 7 significant
digits, double precision contains about 14 significant digits, and
quad precision contains about 28 significant digits
• Consider the following equations:
0.0003𝑥𝑥1 +3𝑥𝑥2 = 1.0002
𝑥𝑥1 + 𝑥𝑥2 = 1

NGN 509 22
Pitfall of Elimination Methods – Roundoff Error

Pivoting – interchange the two equations

NGN 509 23
Pitfall of Elimination Methods – Roundoff Error
(Simple Pivoting)

NGN 509 24
Pitfall of Elimination Methods – Ill-conditioned
• A well-conditioned problem is one in which a small change in any of
the elements of the problem causes only a small change in the
solution of the problem
• An ill-conditioned problem is one in which a small change in any of
the elements of the problem causes a large change in the solution of
the problem
• Since ill-conditioned systems are extremely sensitive to small changes
in the elements of the problem, they are also extremely sensitive to
round-off errors.

NGN 509 25
Pitfall of Elimination Methods – Ill-conditioned

NGN 509 26
Pitfall of Elimination Methods – Ill-conditioned
• With infinite precision arithmetic, ill-conditioning is not a problem.
• With finite precision arithmetic, round-off errors effectively change
the elements of A and b (for Ax=b) slightly, and for ill-conditioned
system, large changes (i.e., errors) can occur in the solution – as
shown in the example.
• Assuming that scaled pivoting has been performed, the only possible
remedy to ill-conditioning is to use higher precision arithmetic
• Several ways to check a matrix A for ill-conditioning; but none of
these approaches is foolproof, and none give a quantitative measure
of ill-conditioning
• Read Sec 1.6.3 (pp55-57) for background theory
NGN 509 27
Direct Elimination
• U represents an upper triangular matrix,
• L is a lower triangular matrix and
• I denotes the identity matrix.
• A square matrix is called triangular if it contains only zero elements
on one side of the leading diagonal

NGN 509 28
Example: Gauss elimination
multiple sets of equations

• The augmented coefficient matrix is

NGN 509 29
Example: Gauss elimination (multiple sets of equations)

NGN 509 30
Example: Gauss elimination (multiple sets of equations)

NGN 509 31
Example: Gauss elimination (multiple sets of equations)

NGN 509 32
 Systems of Linear Algebraic Equations
• System of linear equations can be expressed very compactly in terms
of matrix

• Two approaches to solve the equations

 Direct Methods: Based on algebraic elimination; e.g., Gauss elimination
 Iterative Methods: assume a trial solution, and iterate to convergence; e.g.,
succession-over-relaxation (SOR)
NGN 509 33
Iterative Methods (1.7)
• Iterative, or indirect methods, start with an initial guess of the
solution x, and then repeatedly improve the solution until the change
in x becomes negligible
• Since the required number of iterations can be very large, the indirect
methods are, in general, slower than their direct counterparts
• However, iterative methods do have the following advantages:
• It is feasible to store only the nonzero elements of the coefficient matrix. This
makes it possible to deal with very large matrices that are sparse, but not
necessarily banded.
• Iterative procedures are self-correcting, meaning that roundoff errors (or
even arithmetic mistakes) in one iterative cycle are corrected in subsequent
cycles

NGN 509 34
 Systems of Linear Algebraic Equations
• equation sets arising from physical problems are often very large
• possible to reduce the storage requirements and the run time by
exploiting special properties of the coefficient matrix (e.g., most
elements of a sparse matrix are zero)

• many algorithms dedicated to the solution of large sets of equations,

each one being tailored to a particular form of the coefficient matrix
(symmetric, banded, sparse, etc.).
NGN 509 35
 Special Pattern of Linear Algebraic Equations
• Certain problems give an unique pattern; e.g., tridiagonal system of
equations
• Becomes important consideration when the executing time and/or computer
storage are essential (e.g., differential equations for a very “large” problem
with implicit solving methods)
• Tx = b where T is a tridiagonal matrix

• Sec 1.5 pp49-52 and ex1.7 (Thomas Algorithm)

NGN 509 36
Special Pattern of Linear Algebraic Equations
• Example: Oil Reservoir Simulation – fluid flow in porous media, a
Parabolic PDE

NGN 509 37
Iterative Methods
• Iterative, or indirect methods, start with an initial guess of the
solution x and then repeatedly improve the solution until the change
in x becomes negligible
• A serious drawback of iterative methods is that they do not always
converge to the solution
• convergence is guaranteed only if the coefficient matrix is diagonally
dominant
• The initial guess for x plays no role in determining whether
convergence takes place—if the procedure converges for one starting
vector, it would do so for any starting vector.
• The initial guess affects only the number of iterations that are
required for convergence
NGN 509 38
Iterative Methods
• Iterative methods do not converge for all sets of equations
• Diagonal dominancies a sufficient condition for convergence for any initial
solution vector.
• Diagonally dominant: Absolute value of each major diagonal element is
equal to or large than the sum of absolute values of all other elements in
that row

• Systems that are not diagonally dominant can be rearranged (i.e., by row
interchanges) to make them diagonally dominant.
• The number of iterations required to achieve convergence depends on:
• The dominance of the diagonal coefficients. As the diagonal dominance increases,
the number of iterations required to satisfy the convergence criterion decreases.
• The method of iteration used.
• The initial solution vector.
• The convergence criterion specified. NGN 509 39
Iterative Methods
• Solve by iterative methods; e.g.,
• Jacobi iteration
• Gauss-Seidel iteration
• Successive-over-relaxation (SOR)
• Iterative procedure
• Begin by assuming an initial solution vector 𝒙𝒙(𝟎𝟎)
• The initial solution vector is used to generate an “improved” solution vector
𝑥𝑥 (1)
• Improved solution vector 𝑥𝑥 (1) based on some strategy for reducing the
difference between 𝑥𝑥 (0) and the actual solution vector x.
• This procedure is repeated (i.e., iterated) to convergence.
• The procedure is convergent if each iteration produces
approximations to the solution vector that approach the exact
solution vector NGN 509 40
Iterative Methods

AX = B

n=3 a11x1 + a12x2 + a13x3 = b1

a21x1+ a22x2 + a23x3 = b2
a31x1 + a32x2 + a33x3 = b3

NGN 509 41
Iterative Methods - remarks
• The Jacobi method is sometimes called the method of simultaneous
iteration because all values of 𝑥𝑥𝑖𝑖 are iterated simultaneously; i.e., all values
(𝑘𝑘+1) (𝑘𝑘)
of 𝑥𝑥𝑖𝑖 depend only on the values of 𝑥𝑥𝑖𝑖
• The Gauss-Seidel method is similar to the Jacobi method ,except that the
most recently computed values of all 𝑥𝑥𝑖𝑖 are used in all computation
• Gauss-Seidel method is sometimes called the method of successive iteration
because the most recent values of all 𝑥𝑥𝑖𝑖 are used in all the calculations.
• Gauss-Seidel iteration generally converges faster than Jacobi iteration
• For Successive-over-relaxation SOR, 𝜔𝜔 = 1 gives Gauss-Seidel method
• No good general method for 𝜔𝜔 (relaxation factor) determination

NGN 509 42
Example: Gauss-Seidel method

• Expression of x1, x2 and x3

• Choosing the starting values x1 = x2 = x3 = 0,
• The final results within five decimal places

NGN 509 43
Accuracy & Precision

NGN 509 44
accuracy and convergence
• All nonsingular systems of linear algebraic equations have an exact
solution, which in principle, can be obtained via direct methods
• Round-off errors are unavoidable as the calculations are performed
with finite precision numbers.
• Round-off errors can be minimized by pivoting, as discussed

NGN 509 45
accuracy and convergence
• When solved by iterative methods, the exact solution of a system of linear
algebraic equations is approached asymptotically as the number of
iterations increases.
• When the number of iterations increases without bound, the numerical
solution yields the exact solution within the round-off limit of the
computing device. Such solutions are said to be correct to machine
accuracy.
• Iterative process should be terminated when some type of accuracy
criterion (or criteria) has been satisfied.
• In iterative methods, the term accuracy refers to the number of
significant figures obtained in the calculations, and the term convergence
refers to the point in the iterative process when the desired accuracy is
obtained.

NGN 509 46
Accuracy
• Absolute error and relative error
• Absolute error is defined as
• Absolute error = approximate value - exact value
• Relative error is defined as
• Relative error = absolute error / exact value
• Relative error can be stated directly or as a percentage
• A relative error / accuracy is preferable to use

NGN 509 47
 Convergence
• Convergence of an iterative procedure is achieved when the desired
accuracy criterion (or criteria) is satisfied. Convergence criteria can be
specified in terms of absolute error or relative error.
• Since the exact solution is unknown, the error at any step in the iterative
process is based on the change in the quantity being calculated from one
step to the next
• For the iterative solution of a system of linear algebraic equations, the
(𝑘𝑘+1) 𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒𝑒 (𝑘𝑘+1) (𝑘𝑘)
error, ∆𝑥𝑥𝑖𝑖 = 𝑥𝑥𝑖𝑖 − 𝑥𝑥𝑖𝑖 is approximated by ∆𝑥𝑥𝑖𝑖 = 𝑥𝑥𝑖𝑖 − 𝑥𝑥𝑖𝑖 .

NGN 509 48
Convergence
• At each step in the iterative procedure, some of the residuals may be
near zero while others are still quite large. Therefore, care is needed to
ensure that the desired accuracy of the complete system of equations is
achieved
• Absolute Error criterion:

• Relative error criterion:

NGN 509 49