SE301: Numerical Methods

Topic 1:
Int d ti n to
Introduction t Numerical
N i l methods
th d and
nd Taylor
T l Series
S i
Lectures 1-4:

SE301_Topic1 1
 Lecture 1
Introduction to Numerical Methods

‡ What are NUMERICAL METHODS?

‡ Why do we need them?
‡ Topics covered in SE301.

Reading Assignment: pages 3-10 of text book

SE301_Topic1 2
N
Numerical
ri l MMethods
th d
Numerical Methods:
Algorithms that are used to obtain numerical
solutions of a mathematical problem.
Why do we need them?
1 No analytical solution exists,
1. exists
2. An analytical solution is difficult to obtain
or not practical.
p

SE301_Topic1 3
Wh t ddo we need
What d
Basic Needs in the Numerical Methods:
„ Practical:
can be computed
p in a reasonable amount of time.
„ Accurate:
‡ Good approximate to the true value
‡ I f
Information
ti about
b t the
th approximation
i ti error
(Bounds, error order,… )

SE301_Topic1 4
O tli
Outlines off the
th Course
C r
‡ Taylor
y Theorem ‡ Solution of linear
‡ Number Representation Equations
‡ Solution of nonlinear ‡ Least Squares curve
Equations fitti
fitting
‡ Interpolation
‡ Solution of ordinary
‡ Numerical Differentiation
differential equations
‡ Numerical Integration
‡ Solution of Partial
differential equations

SE301_Topic1 5
S l ti n off N
Solution Nonlinear
nlin r Eq
Equations
ti n
‡ Some simple equations can be solved analytically
x2 + 4x + 3 = 0
− 4 ± 4 2 − 4 (1 )( 3 )
Analytic solution roots =
2 (1 )
x = − 1 and x = −3
‡ Many other equations have no analytical solution

x 9 − 2 x 2 + 5 = 0 ⎫⎪
−x ⎬ No
N analytic
l ti solution
l ti
x = e ⎪⎭

SE301_Topic1 6
Methods for solving Nonlinear
E
Equations
i
o Bisection Method
o Newton-Raphson Method
o Secant Method

SE301_Topic1 7
Solution of Systems of
Li r E
Linear Equations
ti
x1 + x 2 = 3
x1 + 2 x 2 = 5
We can solve it as
x1 = 3 − x 2 , 3 − x2 + 2 x2 = 5
⇒ x 2 = 2, x1 = 3 − 2 = 1
What to do if we have
1000 equations in 1000 unknowns
SE301_Topic1 8
Cr
Cramer’s
r’ RRule
l iis nott pr
practical
ti l
Cramer' s Rule can be used to solve the system
3 1 1 3
5 2 1 5
x1 = = 1, x2 = =2
1 1 1 1
1 2 1 2

But Cramer' s Rule is not practical for large problems.

To solve N equations in N unknowns we need (N + 1)(N − 1)N!
multiplications.
To solve a 30 by 30 system, 2.3 ×1035 multiplications are needed.
A super computer needs more than 10 20 years to compute.
SE301_Topic1 9
Methods for solving Systems of Linear
E
Equations
i
o Naive Gaussian Elimination
o Gaussian Elimination with Scaled
Partial pivoting
o Algorithm for Tri-diagonal
Equations

SE301_Topic1 10
C r Fitti
Curve Fitting
‡ Given a set of data

x 0 1 2
y 0.5 10.3 21.3

‡ Select a curve that best fit the data. One

choice is find the curve so that the sum of
the square of the error is minimized.

SE301_Topic1 11
I t rp l ti
Interpolation
‡ Given a set of data

xi 0 1 2
yi 0.5 10.3 15.3

‡ find a polynomial P(x) whose graph

passes through all tabulated points.
points

yi = P ( xi ) iff xi is in the table

SE301_Topic1 12
M th d fforr C
Methods Curve
r Fitti
Fitting
o Least Squares
o Linear Regression
o Nonlinear least Squares Problems
o Interpolation
o Newton
e to polynomial
po y o a interpolation
te po at o
o Lagrange interpolation

SE301_Topic1 13
I t r ti
Integration
‡ Some functions can be integrated
analytically

3 3
1 2 9 1
∫ xdx = 2 x = 2 − 2 = 4
1 1

But many functions have no analytical solutions

a

∫e
−x2
dx = ?
0

SE301_Topic1 14
M h d ffor N
Methods Numerical
i l IIntegration
i
o Upper and Lower Sums
o Trapezoid Method
o Romberg Method
o Gauss Quadrature

SE301_Topic1 15
Solution of Ordinary Differential Equations
A solution to the differential equation
x(t ) + 3 x (t ) + 3 x(t ) = 0
x (0) = 1; x(0) = 0
is a function x(t)
( ) that satisfies the equations
q

*A
Analytical
l ti l solutions
l ti are available
il bl for
f
special cases only
SE301_Topic1 16
Solution of Partial Differential
E ti
Equations
Partial Differential Equations are more
difficult to solve than ordinary differential
equations

∂ u2
∂ u
2
+ +2=0
∂x 2
∂t 2

i (πx)
u (0, t ) = u (1, t ) = 0, u ( x,0) = sin(

SE301_Topic1 17
S
Summary
r
Topics Covered in the Course
‡ Numerical Methods:
Algorithms that are
‡ Solution of nonlinear Equations
used to obtain ‡ Solution of linear Equations
numerical solution of a
‡ Curve fitting
mathematical problem.
‡ We need them when „ Least Squares
No analytical solution „ Interpolation
exist or it is difficult ‡ Numerical
N me ical Integration
Integ ation
to obtain.
‡ Numerical Differentiation
‡ Solution of ordinary differential
equations
‡ Solution of Partial differential
q
equations

SE301_Topic1 18
 Lecture 2
Number Representation and accurcy
‡ Number Representation
‡ Normalized Floating Point Representation
‡ Significant Digits
‡ Accuracy and Precision
‡ Rounding and Chopping

‡ Reading assignment: Chapter 3

SE301_Topic1 19
R pr
Representing
ti R Reall N
Numbers
b r
‡ You are familiar with the decimal system
y

312.45 = 3 ×10 2 + 1×101 + 2 ×100 + 4 ×10 −1 + 5 ×10 −2

‡ Decimal System Base =10 , Digits(0,1,…9)
‡ Standard Representations

± 3 1 2 . 4 5
sign integral fraction
part part
SE301_Topic1 20
Normalized Floating Point
R
Representation
i
‡ Normalized Floating
g Point Representation
p
± 0. d1 d 2 d 3 d 4 × 10 n
sign
i mantissa
i exponent

d1 ≠ 0, n : integer

‡ No integral part,
‡ Advantage Efficient in representing very small or very large numbers

SE301_Topic1 21
C l l t rE
Calculator Example
pl
‡ suppose you want to compute
3.578 * 2.139
using a calculator with two
two-digit
digit fractions

3 57
3.57 * 2 13 = 7
2.13 7.60
60

True answer 7.653342

SE301_Topic1 22
Bi
Binary System
S

‡ Binary System Base=2, Digits{0,1}

± 0. 1 b2 b3 b4 × 2 n
sign
i mantissa
ti exponentt

b1 ≠ 0 ⇒ b1 = 1

−1 −2 −3
(0.101) 2 = (1× 2 + 0 × 2 + 1× 2 )10 = (0.625)10

SE301_Topic1 23
7-Bit Representation
( i 1 bit,
(sign: bi MMantissa
i 3bits,exponent
3bi 3 bits)
bi )

SE301_Topic1 24
Fact
‡ Number that have finite expansion
p in one numbering
g
system may have an infinite expansion in another
numbering system

(0.1)10 = (0.000110011001100...)) 2
‡ You can never represent 0.1 exactly in any computer

SE301_Topic1 25
R
Representation
i

Hypothetical Machine (real computers use ≥ 23 bit mantissa)

Mantissa
a t ssa 2 b
bits
ts e
exponent
po e t 2 b
bitt sign
sg 1bbitt

Possible machine numbers

.25 .3125 .375 .4375 .5 .625 .75 .875
1 1.25 1.5 1.75

SE301_Topic1 26
 1

08R
Representation
0.8 i
Gap near zero
0.6

0.4

0.2

-0
0.2
2

-0.4

-0.6

-0.8

-1
0 0.2
SE301_Topic1 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2
27
R
Remarks
rk

‡ Numbers that can be exactly represented are called

machine numbers
‡ Difference between machine numbers is not uniform
sum of machine numbers is not necessarily a machine
number
0 25 + .3125
0.25 3125 =0.5625
0 5625 (not
( t a machine
hi number)
b )

SE301_Topic1 28
Si ifi t Di
Significant Digits
it

‡ Significant
g digits
g are those digits
g that can be used with
confidence.

SE301_Topic1 29
 48 9
48.9

SE301_Topic1 30
A r and
Accuracy d Pr
Precision
ii

‡ Accuracy is related to closeness to the true value

‡ Precision is related to the closeness to other estimated

values

SE301_Topic1 31
SE301_Topic1 32
R
Rounding
di andd Chopping
Ch ppi

‡ Rounding: Replace the number by the nearest

machine number
‡ Ch
Chopping:
i Th
Throw all
ll extra di
digits
i

SE301_Topic1 33
Fi 3.7
Fig 37

SE301_Topic1 34
 Error Definitions
T Error
True E

can be computed if the true value is known

Absolute True Error
Et = true value − approximation
Absolute Percent Relative Error
true value − approximation
εt = *100
true value

SE301_Topic1 35
Error Definitions
Estimated error

When the true value is not known

EstimatedAbsoluteError
Ea = current estimate− prevoiusestimate
EstimatedAbsolutePercent RelativeError
current estimate− prevoiusestimate
εa = *100
current estimate

SE301_Topic1 36
Notation

We say
y the estimate is correct to n decimal
digits if −n
Error ≤ 10

We say the estimate is correct to n decimal

digits rounded if 1 −n
Error ≤ × 10
2

SE301_Topic1 37
S
Summary
r
‡ Number Representation
Number that have finite expansion in one numbering system may
have an infinite expansion in another numbering system.
‡ Normalized Floating Point Representation
„ Efficient in representing very small or very large numbers
„ Difference between machine numbers is not uniform
„ Representation error depends on the number of bits used in the
mantissa.

SE301_Topic1 38
 Lectures 33-44
Taylor Theorem
‡ Motivation
‡ T l Th
Taylor Theorem
‡ Examples

Reading assignment: Chapter 4

SE301_Topic1 39
M ti ti
Motivation
‡ We can easily
W il compute expressions
i like
lik
3 × 10 2
2( x + 4)
But, How do you compute 4.1, sin(0.6) ?

We can use the d fi i i to compute

h definition
b
sin(0.6) ? a
0.6
is this a practicalway?

SE301_Topic1 40
T l r SSeries
Taylor ri
Th Taylor
The T l series i off f ( x) about
i expansion b t x0
f ( 2) ( x0 ) f (3) ( x0 )
f ( x0 ) + f ( x0 ) ( x − x0 ) +
'
( x − x0 ) +
2
( x − x0 )3 + ...
2! 3!
or
∞
1 (k )
Taylor Series = ∑ k! f ( x0 ) ( x − x0 ) k

k =0
if the series converge we can write
∞
f ( x) = ∑ k! f ( x0 ) ( x − x0 ) k
1 (k )
k =0

SE301_Topic1 41
Taylor Series
E
Example
l 1

Obtain Taylor series expansion of f ( x) = e x about x = 0

f ( x) = e x f ( 0) = 1
f ' ( x) = e x f ' ( 0) = 1
f ( 2 ) ( x) = e x f ( 2 ) ( 0) = 1
f ( k ) ( x) = e x f ( k ) (0) = 1 for k ≥ 1
∞ ∞ k
x
e x = ∑ f ( k ) ( x0 ) ( x − x0 ) k = ∑
1
k = 0 k! k = 0 k!

The series converges for x < ∞

SE301_Topic1 42
Taylor Series
3

E
Example
l 1
2.5
exp(x)
p( )
1+x+0.5x 2
2

1+x

1.5

1
1

0.5

0
-1
1 -0
0.8
8 -0
0.6
6 -0
0.4
4 -0
0.2
2 0 02
0.2 04
0.4 06
0.6 08
0.8 1

SE301_Topic1 43
Taylor Series
E
Example
l 2

Obtain Taylor series expansion of f ( x) = sin( x) about x = 0

f ( x) = sin(( x) f (0) = 0
f ' ( x) = cos( x) f ' (0) = 1
f ( 2) ( x) = − sin( x) f ( 2) (0) = 0
f (3) ( x) = − cos( x) f (3) (0) = −1
∞
f ( k ) ( x0 ) x 3
x 5
x 7
sin( x) = ∑ k!
( x − x0 ) k = x − +
3! 5! 7!
− + ....
k =0
The
h series f x <∞
i converges for
SE301_Topic1 44
 4

3
x

1 x-x 3/3!+x 5/5!

0 sin(x)

-1
x-x 3/3!

-2

-3

-4
-4
4 -3
3 -2
2 -1
1 0 1 2 3 4

SE301_Topic1 45
Convergence of Taylor Series
(Observations, Example 1)

‡ The Taylor
Th T l seriesi converges fast f (few
(f terms
are needed) when x is near the point of
expansion.
p If ||x-c|| is large
g then more terms
are needed to get good approximation.

SE301_Topic1 46
Taylor Series
E
Example
l 3
1
Obtain Taylor series expansion of f(x) = about x = 0
1− x
1
f ( x) = f (0) = 1
1− x
1
f ' ( x) = f ( 0) = 1
(1 − x )2
2
f ( 2 ) ( x) = f (0) = 2
(1 − x )3
6
f ( 3) ( x ) = f (0) = 6
(1 − x )4
1
Taylor Series Expansion of = 1 + x + x 2 + x 3 + ....
1− x
SE301_Topic1 47
Example 3
remarks
k

‡ C
Can we apply
l Taylor
T l series
i for
f x>1??
1??

‡ How many terms are needed to get good

approximation???

These questions will be answered using

Taylor Theorem

SE301_Topic1 48
T l r Th
Taylor Theorem
r
If a function f(x) posses continuous derivatives
of orders 1,2,..., (n + 1) in a closed interval [a, b]
then for any c ∈ [a, b] (
(n+1)
) terms Truncated
Taylor Series
n
f ( k ) (c )
f ( x) = ∑ k!
( x − c) k + En+1
k =0
Reminder
where
f ( n+1) (ξ )
En+1 = ( x − c) n+1 and ξ is between x and c.
(n + 1)!
(n
SE301_Topic1 49
 Taylor Theorem

We can apply Taylor thorem for

1
f(x) = with point of expansion c = 0 if | x | < 1
1− x

if [a, b] includes x = 1 then the function and its

derivatives are not defined.
defined
⇒ Taylor Theorem is not applicable.

SE301_Topic1 50
Err r T
Error Term
r
T get an idea
To id about
b the
h approximat
i ion
i error

We can derive an upper bound on

n +1)
f (ξ )
((n+
n +1
En+1 = ( x − c)
(n + 1))!
for all values of ξ between x and c.

SE301_Topic1 51
Error Term for
E
Example
l 4

How large is the error if we replaced f ( x) = e x by

the first 4 terms (n = 3) of its Taylor series expansion
about x = 0 when x = 0.2 ?
f ( k ) ( x) = e x f ( k ) (ξ ) ≤ e 0.2 for k ≥ 1
f ( n+1) (ξ )
En+1 = ( x − c) n+1
(n + 1)!
e 0 .2
En+1 ≤ (0.2)n+1 ⇒ E4 ≤ 8.14268E − 05
(n + 1)!
SE301_Topic1 52
Alternative form of Taylor Theorem

L f ( x) have
Let h continuous
ti d i ti s off
derivative
orders 1,2,...(n + 1) on an interval [a,b] ,
and x ∈ [a,b] and x + h ∈ [a,b] then
n
f ( k ) ( x) k
f ( x + h) = ∑ k!
h + En+1
k =0
( n +1)
(ξ ) n+1
f
En+1 = h where ξ is between x and x + h
(n + 1)!
SE301_Topic1 53
Taylor Theorem
Alternative forms

( n +1)
n
f ( k ) (c ) f (ξ )
f ( x) = ∑ k!
k
( x − c) +
(n + 1)!
( x − c) n+1
k =0
where ξ is between x and c

x → x + h, c → x

n
f ( k ) ( x) k f ( n+1) (ξ ) n+1
f ( x + h) = ∑ k!
h +
(n + 1)!
h
k =0
where
h ξ is
i between
b d x+h
x and
SE301_Topic1 54
M
Mean V
Value
l ThTheorem
r
If f(x)
f( ) is
i a continuous
ti f ti on a closed
function l d interval
i t l [a,
[ b]
and its derivative is defined on the open interval (a, b)
then there exist ξ ∈ [a, b]
df(ξ ) f(b) − f(a)
=
dx (b-a)
y Theorem n = 0, x = a, x + h = b
Proof : Use Taylor
df(ξ )
f(b) = f(a) + (b-a)
dx
SE301_Topic1 55
Alt r ti Series
Alternating S ri Th
Theorem
r
Consider the alternating series
S = a1 − a2 + a3 − a4 + "
⎧ a ≥ a ≥ a ≥ a ≥" ⎧The series converges
⎪⎪ 1 2 3 4
⎪
Iff ⎨ and then ⎨ and
⎪ lim a = 0 ⎪ S − S n ≤ an +1
⎪⎩ n→∞ n ⎩
S n : partial sum
s m (sum
(s m of the first n terms)
an +1 : First omitted term

SE301_Topic1 56
Alternating Series
E
Example
pl 5
1 1 1
sin(1) can be computed using sin(1) = 1 − + − + "
3! 5! 7!
This is a convergent alternating series since
a1 ≥ a2 ≥ a3 ≥ a4 ≥ " and lim an = 0
n →∞

Then
⎛ 1⎞ 1
sin(1) − ⎜1 − ≤
⎝ 3! ⎠ 5!
⎛ 1 1⎞ 1
sin(1) − ⎜1 − + ≤
⎝ 3! 5! ⎠ 7!
SE301_Topic1 57
E
Example
pl 6
Obtain the Taylor series expansion
of f(x) = e 2x +1 about c = 0.5 (the center of expansion)
How large can the error be when (n + 1) terms are used
to app ate e 2x +1 w
approximat
o t x = 1?
with

SE301_Topic1 58
E
Example
l 6

Obtain Taylor series expansion of f ( x) = e 2 x +1 , c = 0.5

f ( x) = e 2 x +1 f (0.5) = e 2
f ' ( x) = 2e 2 x +1 f ' (0.5) = 2e 2
f ( 2) ( x) = 4e 2 x +1 f ( 2) (0.5) = 4e 2
f ( k ) ( x) = 2 k e 2 x +1 f ( k ) (0.5) = 2 k e 2
∞
f ( k ) (0.5)
e 2 x +1
= ∑ k!
( x − 0.5) k
k =0
k
( x − 0.5) 2 k 2 ( x − 0.5)
= e + 2e ( x − 0.5) + 4e
2 2 2
+ ... + 2 e + ...
2! k!
SE301_Topic1 59
Example 6
Error term

f ( k ) ( x) = 2 k e 2 x +1
f ( n+1) (ξ )
Error = ( x − 0.5) n+1
(n + 1)!
( x − 0.5) n+1
Error = 2 n+1 e 2ξ +1
(n + 1)!
( x − 0.5) n+1
Error ≤ 2 n +1
max e 2ξ +1
(n + 1)! ξ ∈[0.5,1]
n +1
( x − 0 . 5)
Error ≤ 2 n+1 e3
(n + 1)!
SE301_Topic1 60
R
Remark
rk
‡ In this course all angles are assumed to
be in radian unless you are told otherwise

SE301_Topic1 61
M l ri series
Maclurine ri
Find Maclurine series expansion of cos (x)

Maclurine series is a special case of Taylor

series with the center of expansion c = 0

SE301_Topic1 62
Taylor Series
E
Example
l 7
Obt i Maclurine
Obtain M l i series i off f ( x) = cos(( x)
i expansion

f ( x) = cos( x) f ( 0) = 1
f ' ( x) = − sin( x) f ' (0) = 0
f ( 2 ) ( x) = − cos( x) f ( 2 ) (0) = −1
f ( 3) ( x) = sin( x) f ( 3 ) ( 0) = 0
∞
f ( k ) ( 0) x x x
cos( x) = ∑
2 4 6
( x) k = 1 − + − + ....
k =0 k! 2! 4! 6!
The series converges for x < ∞
SE301_Topic1 63