SOLVING A SYSTEM OF LINEAR EQUATIONS

4.4.1. Gauss Elimination .Used to solve a system of linear equations of the form B is an Mx1 matrix. Ax=B , where A is an MxN Matrix and

The Algorithm .At the k th iteration, the Gauss forward elimination results in: k k1 k1 . amn=amn amk /ak1 ak1 for m ,n=k 1, k1,, M (1) kk kn k k1 k1 k1 . bm =bm a mk /a kk bkk1 for m=k 1, k2,, M ...(2) . After having the triangular matrixvector equation, we find the solutions using backward substitution. . x m=bmm1
n=m1

m1 m1 amn x n/amm for m=M , M 1,, 1 ...(3)

k The core formula used for Gauss elimination requires division by akk 1 at the k th stage, where k k akk1 is the diagonal element in the k th row. What if akk1=0 ? In such a case, it is customary to switch the k th row and another row below it having the element of the largest absolute value in the k th column. This procedure, called partial pivoting .

Read more about this! 4.4.2. LU Decomposition Gaussian elimination with pivoting is the most efficient and accurate way to solve a system of linear equations of the form Ax=b . Most of the work in this method is spent on the matrix A itself. If we need to solve several different systems with the same A , and A is big, then we would like to avoid repeating the steps of Gaussian elimination on A for every different b . This can be accomplished by the LU decomposition, which in effect records the steps of Gaussian elimination. LU decomposition (factorization) of a nonsingular (square) matrix A means expressing the matrix as the multiplication of a lower triangular matrix L and an upper triangular matrix U , where a lower/ upper triangular matrix is a matrix having no nonzero elements above/below the diagonal. For the case where some row switching operation is needed like in the Gauss elimination, we include a permutation matrix P representing the necessary row switching operation(s) to write the LU decomposition as P A=LU The usage of the permutation matrix P is exemplified by: 0 0 1 a11 a12 a 13 a31 a 32 a33 PA= 1 0 0 a21 a22 a 23 = a11 a 12 a13 0 1 0 a31 a32 a 33 a21 a 22 a23

Method1: To take a close look at the LU decomposition, we consider a 3 3 nonsingular matrix: a 11 a12 a13 1 0 0 u11 u12 u 13 a 21 a22 a23 = l 21 1 0 0 u22 u 23 a 31 a32 a33 l 31 l 32 1 0 0 u 33

a 11 a12 a13 u11 u12 u13 = l 21 u11 l 21 u12u 22 a 21 a22 a23 l 21 u 13u23 a 31 a32 a33 l 31 u11 l 31 u12l 32 u 22 l 31 u13l 32 u23u 33

First, equating the first rows of both sides yields: u1n=a1n , n=1, 2,3 Then, equating the second rows of both sides yields: a 21=l 21 u11 , a22 =l 21 u12 u22 , a 23=l 21 u13u 23 from which we can get: l 21=a21 /u11 ,u22=a21 l 21 u12 , u23=a23 l 21 u13 Now, equating the third rows of both sides yields: a31=l 31 u 11 , a32 =l 31 u 12l 32 u22 , a33 =l 31 u 13 u32 u23u33 from which we can get l 31=a31 /u11 , l 32= a32 l 31 u 12/u22 , u33=a33 l 31 u13 l 32 u 23 From these, we determine the two matrices L and U . Method 2 The main idea of the LU decomposition is to record the steps used in Gaussian elimination on A in the places where the zero is produced. Consider the matrix: 1 2 3 A= 2 5 12 0 2 10 The first step of Gaussian elimination is to subtract 2 times the first row from the second row. In order to record what we have done, we will put the multiplier, 2, into the place it was used to make a zero, i.e. the second row, first column. In order to make it clear that it is a record of the step and not an element of A, we will put it in parentheses. This leads to: 1 2 3 A= 2 1 6 0 2 10 There is already a zero in the lower left corner, so we dont need to eliminate anything there. We record this fact with a (0). To eliminate the third row, second column, we need to subtract 2 times the second row from the third row. Recording the 2 in the spot it was used we have: 1 2 3 2 1 6 0 2 2 Let U be the upper triangular matrix produced, and let L be the lower triangular matrix with the records and ones on the diagonal, i.e.:

1 0 0 L= 2 1 0 , 0 2 1 then we have the following mysterious coincidence:

1 2 3 U = 0 1 6 0 0 2 LU = A

Using LU to solve equations If we also include pivoting, then an LU decomposition for A consists of three matrices P , L and U

such that:

P A=LU To use the information P , L , U to solve a linear system multiplying by the pivot matrix:
P Ax=P b d
Substituting LU for P A we get:

Ax=b we first pivot both sides by

LU x=d Ly=d

Then we need only to solve two back substitution problems: and

U x= y. Example: Find the solution of the system of linear equations whose A and b are given as: 2 3 1 5 A= 4 4 3 , b= 3 2 3 1 1

Solution: After the first Gauss elimination step, (b)=(b)-2(a) and (c)=(c)-(-a) 2 3 1 A= 2 2 1 1 6 2

After the second Gauss elimination step, (c)=(c)-(-3)(a) 2 3 1 A= 2 2 1 1 3 5

From this, we can easily find the matrices L and U 1 0 0 2 3 1 1 0 0 L= 2 U= 0 2 1 and since no row exchanges were done, P= 0 1 0 1 0 and 1 3 1 0 0 5 0 1 1 PA=LU

To solve the system of linear equations: forward substitution Ly=Pb=b y1=5 2y1 y2=3 y2=7 y13y2 y3=1 y3=1521=15 backward substitution Ux= y 5x3=15 x3=3 2x2x3=7 x2=2 2x13x2x3=5 x1=1

4.5 Iterative Methods

In these methods, an initial estimate is given for the parameters to be solved, and the equations are solved, giving an updated version of the parameters. These new values are then inserted back into the equations and the process continues until the desired solution is reached. The two iterative methods discussed here are the Jacobi method and the Gauss-Seidel method.

4.5.1.

Jacob Method
An algorithm for determining the solutions of a system of linear equations iteratively Given a square system of n linear equations Ax=b where

Then A can be decomposed into a diagonal component D, and the remainder R: A = D + R where

The system of linear equations may be written as DR x=b D xR x=b , and finally, D x=b R x 1 Eqn (1) results in the following equivalent system of linear equations a11 x 1=b 1{a12 x 2a13 x 3a1n x n } a22 x 2=b 2{a21 x 1a23 x 3a2n x n } 2 a nn x n=b n{an1 x 1an2 x 2a 1 {n1 } x n1 } The Jacobi method solves the left hand side of eqn (1) or eqn (2) for x using previous values of x on the right hand side. k 1 1 1 k x =D bD R x Or the element-based formula is given by xik1= 1 b a x k a ii i ij j ji

Algorithm Choose an initial guess x0 to the solution while convergence not reached do for i=1 step until n do =0 for j=1 step until n do if j !=i then =aij x k1 j end if end (j-loop) bi k xi = aii end (i-loop) check if convergence is reached end (while convergence condition not reached loop)

The iteration process results in the vector sequence x 0 , x 1 ,x k . If the matrix A is strictly diagonally dominant, the sequence converges to the exact solution x*. => a ii a ij for each row i
ji

The Jacobi method sometimes converges even if these conditions are not satisfied The stop criterion in the algorithm can be given by: if x k x k1 then x* = x k with accuracy Example 1: Perform 5 iterations using the Jacobi method for the system given below. Work with an intermediate accuracy of five digits after the decimal comma, and for initial guess choose the zero vector x 0=0,0,0 . 4 x1 x2 =2 x1 + 4 x2 x3 = 6 x2 + 4 x3 = 2 Solution 4 1 0 A= 1 4 1 0 1 4

2 b= 6 2 0 x 0 = 0 0

Formula: x k 1 =D1 bD1 R x k 1 1 D b=d and D R=C 0.5 0 0.25 0 d = 1.5 and C= 0.25 0 0.25 0.5 0 0.25 0 k 1 k x =d C x Convergence check:

3 <4 and 2 < 4 and 3 < 4 => The Jacobi method will converge Iteration 1 0 x =dC x 0.5 0 0.25 0 0 0.5 x 1= 1.5 0.25 0 0.25 0 = 1.5 0.5 0 0.25 0 0 0.5

x 2 =d C x 1 0.5 0 0.25 0 0.5 0.875 2 x = 1.5 0.25 1.5 = 1.75 0 0.25 0.5 0 0.25 0 0.5 0.875

x 3 =d C x 2 0.5 0 0.25 0 0.875 0.9375 3 x = 1.5 0.25 0 0.25 1.75 = 1.9375 0.5 0 0.25 0 0.875 0.9375

x 4 =dC x 3 0.5 0 0.25 0 0.9375 0.98438 4 x = 1.5 0.25 0 0.25 1.9375 = 1.96875 0.5 0 0.25 0 0.9375 0.98438
5

x =dC x 0.5 0 0.25 0 0.98438 0.99219 x 5= 1.5 0.25 0 0.25 1.96875 = 1.99219 0.5 0 0.25 0 0.98438 0.99219 At the 13th iteration, 1.00000 x 13=2.00000 1.00000

A x=b

Home Work: 1. Use Jacobi method to find the solution of a linear system of the form with initial x 0 given by A= 2 1 , b= 11 and x 0= 1 5 7 13 1 The solution after 25 iterations is: x= 7.111 3.222

2. Use the Jacobi method to solve the following system of linear equations 7x 13x 2x 3=18 2.85997 6 2 x 1 9x 24 x 3=12 Answer: At x = 0.68261 x 1 4 x 212 x 3 =6 0.03365

4.5.2.

Gauss-Seidel Method
An iterative method used to solve systems of linear equations Similar to the Jacobi method Given a square system of n linear equations A x=b where A , x and b are given as in the Jacobi method A can be decomposed into a lower triangular component L and a strictly upper triangular component U A= LU where

The system of linear equations may be rewritten as: L x=bU x 3 Eqn (3) results in the following equivalent system of linear equations a11 x1 =b1{a12 x 2a13 x 3a 1n x n } a 21 x 1a22 x 2=b2{a23 x3 a24 x 4a 2n x n } a n1 x 1an2 x 2a n3 x 3ann x n=bn The GaussSeidel method, like the Jacobi method, solves the left hand side of this expression for x, using previous value for x on the right hand side. k 1 1 k x =L bU x However, by taking advantage of the triangular form of L , the elements x k 1 can be computed sequentially using forward substitution. The element-based formula is given by x k1= i 1 b i a ij xjk aij x jk1 ,i=1, 2,n aii ji ji

Algorithm Inputs: A , b Output: x Choose an initial guess x 0 repeat until convergence to the solution

for i from 1 until n do =0 for j from 1 until i 1 do =aij x k1 j end (j-loop)

for j from i + 1 until n do =aij x k j end (j-loop) 1 k1 x i = bi a ii end (i-loop) check if convergence is reached end (repeat)

The iteration process results in the vector sequence x 0 , x 1 ,x k . If the matrix A is strictly diagonally dominant, the sequence converges to the exact solution x*. => a ii a ij for each row i
ji

The GaussSeidel method sometimes converges even if these conditions are not satisfied. The stop criterion in the algorithm can be given by: if x k x k1 then x* = x k with accuracy Example 1 A linear system shown as A x=b is given by: 3 11 A= 16 and b= 7 11 13 0 1 Use an initial estimate x = to solve for x using the Gauss-Seidel method. 1 Solution: k 1 k x =dC x where C=L1 U and d =L1 b 16 0 0 0 L= , U= 7 11 0 3

T = 0 0.1875 , C=0.6875 0.7443 0 0.1193 Convergence check 16 > 3 and 11 > 7 => The iterative process will converge to the solution Iterations

The iteration converges to

x=

0.8122 0.6650

Homework 1. A system of linear equations shown as A x=b is given by 2 3 11 A= and b= 5 7 13 0 1.1 use x = to find the solution of the system using the Gauss-Seidel method. 2.3 [check for convergence] 2. Solve the following system of linear equations using the Gauss-Seidel method 7x 13x 2x 3=18 2 x 1 9x 24 x 3=12 x 1 4 x 212 x 3 =6 Answer: 2.85957 4 At x = 0.68273 0.03412