Lecture05 and 06

Advanced Algorithms Design and
Analysis
By
Nazir Ahmad Zafar
Dr Nazir A. Zafar
Advanced Algorithms Design and Analysis
Lecture No 5
Further Techniques Solving

Recurrence Relations
(Algorithms Analysis Techniques)
Dr Nazir A. Zafar
Advanced Algorithms Design and Analysis
Genralization: Previous Lecture

If a recurrence is of the form
a0t n + a1t n 1 + . . . + ak t n k = b1n p1 (n) + b2n p2 (n) + ...

Then the characteristics polynomial is
(a0 x + a1 x
k
k 1
+ . . . + ak )( x b1 )
d1 +1
( x b2 )
d 2 +1
...,
It has one factor for left hand side for homogenous

part, and factor corresponding to each term on right
hand side for non-homogenous part.
Once the characteristics polynomial is obtained the
recurrence can be solved as before.
Dr Nazir A. Zafar
Advanced Algorithms Analysis and Design
Today Covered
In this lecture following will be covered
Assumption in solving recurrence
The substitution Method
The recursion tree Method
The Master Theorem
Conclusion
Dr Nazir A. Zafar
Assumption in Solving Recurrence Relation

Neglect certain technical details solve recurrences
Assume integer arguments to functions
Because running time T (n) is always defined
when n is an integer
Consequently, for convenience, we shall omit
statements of boundary conditions of recurrences
and assume that T (n) is constant for small n
Recurrence for worst-case time of MERGE-SORT
if n = 1
(1)
T ( n) = n
n
T 2 + T 2 + (n) otherwise

Dr Nazir A. Zafar
Assumption in Solving Recurrence Relation

We do not give any explicit value for small n
Because changing n, T (1) changes solution to
recurrence. Solution typically doesnt change by more
than a constant factor, so order of growth is unchanged
Solve recurrences, often omit floors, ceilings, etc.
First analyze without these details and later determine
whether such assumptions matter or not.
Usually it dose not matter but it is important to know
when it does and when it does not
if n = 1
(1)
T ( n) =
n
2.T ( 2 ) + (n) otherwise
Dr Nazir A. Zafar
Methods Solving Recurrence Relation
The Substitution Method
Dr Nazir A. Zafar
Substitution Method
Substitution method has two steps
Guess the form of the solution
Use mathematical induction to find constants and
show that the solution does work
The name Substitution comes from the substitution
of guessed answer for the function when the
inductive hypothesis is applied to smaller values.
Method is powerful, but it can be applied only in
cases when it is easy to guess the form of answer
The substitution method can be used to establish
either upper or lower bounds on a recurrence.
Dr Nazir A. Zafar
Some Important Rules used in this Section

Prove that
log a b log b c = log a c
Proof
let us suppose that
log = t
log = s
c
b
b
a
a = b and b = c
s
Now b t = c,
a = c,
st
s t = log ca ,
Dr Nazir A. Zafar
(a s ) t = c
log ca = s t
log ba log bc = log ca proved
Some Important Rules used in this Section

Prove that
Proof
log 4n
log 4n
=n
=n
log 34
log 3 (3
log 4n
log 34
) = log 3 (n
log log = log log

n
4
3
3
3
4
log = log log

n
4
3
4
log = log
n
4
Dr Nazir A. Zafar
n
4
log 34
n
3
n
3
log log = log

b
a
c
b
c
a

Solve the recurrence relation given below.
if n = 1
1
T ( n) =
n
3 T ( 4 ) + n otherwise
Solution:
n
(1) T (n) = 3 T ( ) + n
4
replace the value of n by n 4 in (1)

n
n
n
(2) T ( ) = 3 T ( 2 ) +
4
4
4
Dr Nazir A. Zafar

n
n
n
(3) T ( 2 ) = 3 T ( 3 ) + 2
4
4
4
(4) T (
n
4
) = 3T (
k 1
and so on
n
n
)
+
4k
4 k 1
n
Now substitute the value of T ( ) from (2) to (1)
4
n
n
(5) T (n) = 33T ( 2 ) + + n

4
4
n
3
= 32 T ( 2 ) + ( ) n + n
4
4
substitute the value of T (
n
) from (3) to (5) equation, we have
2
4
n
n
3
T ( n) = 3 3 T ( 3 ) + 2 + ( ) n + n
4
4
4
Dr Nazir A. Zafar

n
3
2 n
T ( n) = 3 T ( 3 ) + 3 2 + n + n
4
4
4
3
3 k 1
3 k 2
3
3 0
n
T ( n) = 3 T ( k ) + ( ) n + ( ) + + ( ) n + ( ) n
4
4
4
4
4
k
Let us suppose that n can be expressed as n = 4 k

3
3 2
3 k 1
n
T (n) = 3 T ( ) + n 1 + ( ) + ( ) + .... + ( )
4
4
4
n
3 k
1 ( 4)
k
T (n) = 3 T (1) + n 1 (
3
3k
T (n) = 3 1 + 4n(1 k )
4
k
Dr Nazir A. Zafar
1 + x + x + ... + x
2
k 1
1 xk
)
= 1 (
1 x
T (1) = 1

k
3
3
k
T (n) = 3 + 4n (1 k ) = 3 + 4n (1 )
4
n
k
n3
= 3 + 4n(
)
n
k
= 3 + 4(n 3 ) = 1 3 + 4n 4 3
k
= 4 n 3 3 = 4n 3 3
log n
4
4 k = n,
Dr Nazir A. Zafar
k = log n4

T ( n) = 4n 3 3
log n4
T ( n) = 4n 3 n
T ( n) = 4n 3 n
T ( n) = 4n 3 n
log 34
0 1
Hence T (n) (n)
Dr Nazir A. Zafar
let = log 34
A Hard Example
Solve the recurrecnce relation given below
if n = 1
1
T ( n) =
n
aT ( b ) + n otherwise
Solution :
where 0 < a < b
n
n
n
T (n) = aT ( ) + n = a a T ( 2 ) + a + n
b
b
b
n
a
n
n a
2
= a T ( 2 ) + ( )n + n = a aT ( 3 ) + 2 + ( )n + n
b
b
b
b b
n
a 2
a
3
= a T ( 3 ) + ( ) n + ( )n + n
b
b
b
2
Dr Nazir A. Zafar
A Hard Example
Continuing this process we get
n
a k 1
a k 2
a 1
k
T (n) = a T ( k ) + ( ) n + ( ) n + .... + ( ) n + n
b
b
b
b
n
a
a 2
a k 1
k
= a T ( k ) + n 1 + ( ) + ( ) + .... + ( )
b
b
b
b
Let us suppose that, n = b
a
a k 1
T (n) = a T (1) + n 1 + ( ) + .... + ( )

b
b
Dr Nazir A. Zafar
A Hard Example
a k
1 ( )
k
b
a
k
k
b
= a T (1) + n
)(1 k )
= a T (1) + n(
ba
b
1 a
b
b a
= a 1 + n(
)(
)
k
ba
b
k
b
b a
= a + n(
)(
)
ba
n
k
Dr Nazir A. Zafar
n = b
A Hard Example
b
k
k
)(b a )
= a +(
ba
k
b
b
k
k
=a +
b
a
ba
ba
k
b
ba b k
k
)a
=
b + (
ba
ba
Dr Nazir A. Zafar
A Hard Example
b
a
log bn
k
)b
a
T ( n) = (
ba
ba
let = log
b
a
log ba
k
=
b
n
ba
ba
a
b
a < b log < 1 0 < < 1

a
b
b
a
T ( n) =
n
n T ( n) ( n)
ba
ba
Dr Nazir A. Zafar
More Hard Example using Substitution Method

Solve the recurrence relation given below.
if n = 1
1
T ( n) =
n
2
3 T ( 4 ) + cn otherwise
Solution:
n
2
(1) T (n) = 3 T ( ) + cn
4
n
n
n 2
(2) T ( ) = 3 T ( 2 ) + c( )
4
4
4
Dr Nazir A. Zafar
Contd..
n
n
n 2
(3) T ( 2 ) = 3 T ( 3 ) + c( 2 )
4
4
4
n
n
n 2
(4) T ( 3 ) = 3 T ( 4 ) + c( 3 )
4
4
4
and so on
n
n
n 2
(5) T ( k 1 ) = 3 T ( k ) + c( k 1 )
4
4
4
Dr Nazir A. Zafar
Contd..
n
Now substitute the value of T ( ) from (2) to (1)
4
n
n 2
2
(6) T (n) = 33T ( 2 ) + c( ) + cn
4
4
n
n 2
2
= 3 T ( 2 ) + 3c( ) + cn
4
4
2
n
substitute the value of T ( 2 ) from (3) to (6), we have
4
n
n 2
n 2
T ( n) = 3 3T ( 3 ) + c( 2 ) + 3c ( ) + cn 2
4
4
4
Dr Nazir A. Zafar
Contd..
n
n 2
n 2
2
2
T (n) = 3 T ( 3 ) + 3 c( 2 ) + 3c ( ) + cn
4
4
4
3
n
3 2 2
3 1 2
2
T (n) = 3 T ( 3 ) + ( 2 ) cn + ( 2 ) cn + cn
4
4
4
3
After continuing this process, we get

n
3 k 2
3 1
3 0 2
3 k 1
= 3 T ( k ) + ( 2 ) + ( 2 ) + +( 2 ) + ( 2 ) cn
4
4
4
4
4
k
Dr Nazir A. Zafar
Contd..
Let us suppose that n can be expressed as n = 4
3
3 2
3 k 1
T (n) = 3 T (1) + cn 1 + + ( ) + +( )
16
16 16
3 k
1 ( )
16
T (n) = 3 k 1 + cn 2 (
3
1
16
k
3
16
2
k
= 3 + cn (1 k )
13
16
Dr Nazir A. Zafar
Contd..
since 4 = n (4 ) = n
k
(4 ) = n
2 k
16 = n
k
16
3
T (n) = 3 + cn (1 k )
13
16
k
2
k
n
16
3
= 3 k + cn 2 (
)
2
13
n
16
2
k
= 3 + c (n 3 )
13
k
16
16
2
k
= c n + (1 c)3
13
13
Dr Nazir A. Zafar
Contd..
16 2
16 log n
T (n) = cn + (1 c)3 4
13
13
16 2
16
log 3
=
cn + (1
c)n 4
13
13
let log 34 =
where 0 < < 1
16
16 2
T (n) = cn + (1 c)n
13
13
Hence T (n) (n )
2
Dr Nazir A. Zafar
Observations
if n = 1
1
T ( n) =
n
k
+
(
)
otherwise
a
T
cn
= en + fn
k
log a
b
= en + fn
k
log b 1
b
= en + f .n
k
= ( n
Dr Nazir A. Zafar
k1 . log b
b
max( k , k1 )
suppose b = a
k1
= en + f .n
k
k1
)
Recursion Tree Method
Dr Nazir A. Zafar

Although substitution method can provide a
sufficient proof that a solution to a recurrence is
correct, sometimes difficult to give a good guess.
Drawing out a recursion tree, is a straight forward
way to devise a good guess.
In recursion tree, nodes represent costs of a subproblems in the set of recursive function invocations
We sum costs within each level of the tree to obtain
a set of per-level costs
And then we sum all per-level costs to determine
the total cost of all levels of the recursion.
Recursion trees particularly useful when recurrence
describes running time of divide and conquer algos.
Dr Nazir A. Zafar

Recursion tree is best one used to generate a good
guess, which is then verified by substitution method
When using a recursion tree to generate a good
guess, we can often tolerate a small amount of
sloppiness since we have to verify it later on
If we are careful when drawing out a recursion tree
and summing costs, then we can use a recursion
tree as a direct proof of a solution to any
recurrence of any problem
Here, we will use recursion trees directly to prove
theorem that forms the basis of the master method
Dr Nazir A. Zafar

Example:
Solve the following recurrence using recurrence tree method
(1)
T ( n) =
n
3.T ( ) + (n 2 )
4
if n = 1
if otherwise
Solution: The above recurrence can be written in the form
1
T ( n) =
n
3.T ( ) + cn 2
4
if n = 1
if otherwise
Assumption: We assume that n is exact power of 4.

The recurrence tree is given in the next slide
Dr Nazir A. Zafar

c.n2
T(n) = 3.T(n/4)+c.n2
T(n/4)
T(n/4)
T(n/4)
c.n2
c.(n/4)2
c.(n/4)2
T(n/16) T(n/16)
Dr Nazir A. Zafar
T(n/16)
T(n/16) T(n/16)
c.(n/4)2
T(n/16)
T(n/16) T(n/16)
T(n/16)

Suppose n=4k
c.(n/4)2
c.n2
c.(n/4)2
c.n2
c.(n/4)2
2
c(n/16)2 c(n/16)2 c(n/16)2 c(n/16)2 c(n/16)2 c(n/16)2 c(n/16)2 c(n/16)2 c(n/16)
(3/16).c.n2
(3/16)2.c.n2
T(n/4k) T(n/4k) T(n/4k) T(n/4k) T(n/4k) T(n/4k) T(n/4k) T(n/4k)

Dr Nazir A. Zafar
Log4n
Suppose n = 4
c.n2
c.n2
c.(n/4)2
c.(n/4)2
c.(n/4)2
(3/16).c.n2
2
2
2
2
2
c(n/16)2 c(n/16)2 c(n/16) c(n/16)2 c(n/16)2 c(n/16) c(n/16) c(n/16) c(n/16)
(3/16)2.c.n2
(3/42)k-1 cn2
T(n/4k)
T(1)
T(1)
Dr Nazir A. Zafar
T(n/4k)
T(1)
T(n/4k)
T(1)
T(1)
T(n/4k)
T(1)
T(n/4k)
T(1)
T(n/4k)
T(1) T(1)
T(n/4k)
T(1)
T(1)
T(n/4k)
T(1)
(nlog43)

Now the total computation cost would be
= Cost of Childs + Cost of tree excluding childes
= Cost of Child x total number of Childs + Cost of
all levels excluding childes level
= total number of Childs + sum of costs at each
level excluding childes level.
T ( n) = 3
log 4n
T ( n ) = ( n
Dr Nazir A. Zafar
+ cost at Levels above child level
log 3 4
3 k 1 2
3 1
3 0
) + ( 2 ) + ( 2 ) + +( 2 ) cn
4
4
4

Now total computational cost can be calculated as
T ( n ) = ( n
log 3 4
3 1
3 k 1 2
3 0
) + ( 2 ) + ( 2 ) + +( 2 ) cn
4
4
4
Where 4 = n k = log 4 n
k
T ( n ) ( n
log 3 4
T ( n ) ( n
3 1
3 0
2
) + ( 2 ) + ( 2 ) + cn
4
4
log 3 4
)+(
3
1 ( )
16
)cn = (n
2
log 3 4
16 2
) + cn
13
Hence T (n) = (n 2 )
Dr Nazir A. Zafar
Master Theorem
Dr Nazir A. Zafar
Master Theorem
Lemma 1:
Let a 1, b > 1 be constants, f(n) a non-negative
function defined on exact power of b by recurrence
(1)
T ( n) =
n
a.T ( ) + f (n)
b
if n = 1
if n = b i
where i is a positive integer. Then
T ( n ) = ( n
log b a
)+
log b n 1
j =0
Dr Nazir A. Zafar
n
a .f ( j )
b
j
Master Theorem
Lemma 2: Let a 1, b > 1 be constants, let f(n) be a
non-negative function defined on exact power of b. A
function g(n) defined over exact powers of b by
g ( n) =
log b n 1
j =0
n
a .f ( j )
b
j
can be bounded asymptotically for exact powers of b as

log b a
), for some constants > 0, then g (n) = (n )

1. If f (n) = (n
log
log
(
)
(
)
,
then
(
)
(
. lg n )
f
n
=
n
g
n
=
n
2. If
3. If a.f(n/b) c.f(n), for some constant c < 1 and for all
n b, then g(n) = (f(n))
log b a
Dr Nazir A. Zafar
Master Theorem
Proof:
log b a
)
Case 1: f ( n) = ( n
Which implies that
n
n logb a
)
f ( j ) = (( j )
b
b
We are given that: g (n) =
log b n 1
j =0
n
a .f ( j )
b
j
(1)
(2)
Substituting value from Equation (1) in Equation (2)
g ( n) = (
log b n 1
j =0
Dr Nazir A. Zafar
n logb a
)
a .( j )
b
j
Master Theorem
log b n 1
Consider
j =0
n logb
log b
a .( j )
=n
.
b
a
b
logb a
Assume that:
log b log b
a
j =0
a.b
b
logb a
=x
a
Taking log on both sides: log b (

logb a
log b n 1
logb a
) = log b x
= log b x log b log b . log b = log b x

a
log b log b = log b x 0 = log b x x = 1

a
Dr Nazir A. Zafar
Master Theorem
log b 1
n
j =0
n logb
j
a .( j )
b
log b 1
n
=n
log b
j =0
=n
log b a
a.b
b
logb a
) =n
j
log b a
log b n 1
j
b
(
)
j =0
log b n 1
.((b ) + (b ) + ... + (b )
It is a geometric series with first term 1, common

ratio b and number of terms as logbn. Hence g(n),
=n
log b
= n logb
. log b b
n
1
(b )
1
b
1
log b
log b
(
)
(
)=n
(
)= n
b 1
b 1
b 1
log b n
. log b n
n 1
log b a
log b a
log b a
( ) = c.n
c.n
= ( n )
b 1
Hence proved
Dr Nazir A. Zafar
Master Theorem
Case 2:
f ( n ) = ( n
Which implies that:
log b a
n
n logb a
f ( j ) = (( j ) )
b
b
log b n 1
j =0
n
a .f ( j )
b
j
(3)
(4)
Substituting value from Equation (3) in Equation (4)
g ( n ) = (
Dr Nazir A. Zafar
log b n 1
j =0
n logb a
a .( j ) )
b
j
Master Theorem
log b 1
n
g ( n ) = (
j =0
n logb a
j
a .( j ) )
b
= ( n
log b a
a
b
g ( n ) = ( n
log b
= (n
1)
j =0
= ( n
Dr Nazir A. Zafar
log b a
j =0
We have already proved that

log b n 1
log b n 1
logb a
log b a
(
b
logb a
) )
=1
. (1 + 1 + ... + 1 )
log b n number of terms
. log b )
case is proved
Master Theorem
Case 3:
n
n
c
Given that: a. f ( ) c. f (n) f ( ) . f (n)
b
a
b
n
c 2
n
c
n
c 2
f ( 2 ) . f ( ) ( ) . f ( n) f ( 2 ) ( ) . f ( n)
b
a
b
a
b
a
n
c 3
n
c 2 n
c 3
f ( 3 ) ( ) . f ( ) ( ) . f ( n) f ( 3 ) ( ) . f ( n)
b
a
b
a
b
a
n
c j
In general: f ( j ) ( ) . f (n)
b
a
n
j
Equivalently: a . f ( j ) c j . f (n)
b
Dr Nazir A. Zafar
Master Theorem
log b n 1
n
a .f ( j )
b
j
j =0
(5)
n
j
We have proved that: a . f ( j ) c . f (n)
b
j
(6)
From equation (5) and (6)
g ( n) =
log b n 1
j =0
n
a .f ( j )
b
j
log b n 1
j =0
1
g (n) = f (n)(
) = ( f (n))
1 c
Dr Nazir A. Zafar
j
c
f
n
.
(
)
c . f ( n)
j =0
(7)
Master Theorem
Since f(n) appears in definition of g(n) and all
terms of g(n) are non-negative, we can conclude
easily that
g (n) = ( f (n))
(8)
We have already proved, equation (7), that
g (n) = ( f (n))
From Equations (8) and (9)
Hence it proves the lemma
Dr Nazir A. Zafar
(9)
g (n) = ( f (n))
Master Theorem
Lemma 3 Let a 1, b > 1 be constants, let f(n) be a nonnegative function defined on exact power of b. Define T(n) on
exact powers of b by the recurrence
(1)
T ( n) =
n
a.T ( ) + f (n)
b
if n = 1
if n = b i
where is a positive integer. Then T(n) can be bounded

asymptotically for exact powers of b as
log b a
), for some constants > 0, then T (n) = (n )

1. If f (n) = (n
a
log b
log b a
2. If f (n) = (n ), then T (n) = (n . lg n )
3. If f (n) = (n log ) for some > 0, and a.f(n/b) c.f(n) for some
constant c < 1and sufficiently large n, then T(n) = (f(n))
log b a
a +
Dr Nazir A. Zafar
Master Theorem
Proof: Case 1
Given that
(1)
T ( n) =
n
a.T ( ) + f (n)
b
By Lemma 4.2: T (n) = (n
log b a
)+
if n = 1
if n = b i
log b n 1
j =0
log b a
) + ( n
n
a .f ( j )
b
j
log b a
Hence for case 1 it is proved

Dr Nazir A. Zafar
Master Theorem
Case 2
Again given that
(1)
T ( n) =
n
a.T ( ) + f (n)
b
log b a
)+
log b n 1
j =0
log b a
) + ( n
if n = 1
if n = b i
n
a .f ( j )
b
j
log b a
. lg n)
Hence for case 2 it is also proved

Dr Nazir A. Zafar
Master Theorem
Case 3
log b a
)+
log b n 1
j =0
By Lemma 4.3: T ( n) = ( n
log b a +
Since
f ( n) = ( n
Hence
T (n) = ( f (n))
log b a
n
a .f ( j )
b
j
) + ( f (n))
This proves the Lemma 3

Dr Nazir A. Zafar
Conclusion
First Order homogenous linear recurrence is in fact

a geometric sequence
We studied characteristics of second and higher
order linear homogenous recurrence relations with
constant coefficients
Then non-homogenous recurrence relations with
constant coefficients
Different methods to solve such recurrences
As most of the algorithms are recursive therefore
to give analysis of such algorithms, we have
discussed this powerful technique.
Dr Nazir A. Zafar

Lecture05 and 06

Încărcat de

Informații document

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Lecture05 and 06

Încărcat de

Drepturi de autor:

Formate disponibile

Advanced Algorithms Design and

Advanced Algorithms Design and Analysis

Further Techniques Solving

Advanced Algorithms Design and Analysis

Genralization: Previous Lecture

a0t n + a1t n 1 + . . . + ak t n k = b1n p1 (n) + b2n p2 (n) + ...

It has one factor for left hand side for homogenous

Advanced Algorithms Analysis and Design

Advanced Algorithms Analysis and Design

Assumption in Solving Recurrence Relation

Advanced Algorithms Analysis and Design

Assumption in Solving Recurrence Relation

Advanced Algorithms Analysis and Design

Methods Solving Recurrence Relation

The Substitution Method

Advanced Algorithms Analysis and Design

Advanced Algorithms Analysis and Design

Some Important Rules used in this Section

log a b log b c = log a c

Some Important Rules used in this Section

log log = log log

log = log log

log log = log

Advanced Algorithms Analysis and Design

The Substitution Method

replace the value of n by n 4 in (1)

Advanced Algorithms Analysis and Design

The Substitution Method

(5) T (n) = 33T ( 2 ) + + n

Advanced Algorithms Analysis and Design

The Substitution Method

Let us suppose that n can be expressed as n = 4 k

The Substitution Method

Advanced Algorithms Analysis and Design

The Substitution Method

Hence T (n) (n)

Advanced Algorithms Analysis and Design

where 0 < a < b

Advanced Algorithms Analysis and Design

Let us suppose that, n = b

T (n) = a T (1) + n 1 + ( ) + .... + ( )

Advanced Algorithms Analysis and Design

Advanced Algorithms Analysis and Design

Advanced Algorithms Analysis and Design

a < b log < 1 0 < < 1

Advanced Algorithms Analysis and Design

More Hard Example using Substitution Method

Advanced Algorithms Analysis and Design

Advanced Algorithms Analysis and Design

Advanced Algorithms Analysis and Design

After continuing this process, we get

Advanced Algorithms Analysis and Design

Let us suppose that n can be expressed as n = 4

Advanced Algorithms Analysis and Design

Advanced Algorithms Analysis and Design

where 0 < < 1

Advanced Algorithms Analysis and Design

Recursion Tree Method

Advanced Algorithms Analysis and Design

Recursion Tree Method

Advanced Algorithms Analysis and Design

Recursion Tree Method

Advanced Algorithms Analysis and Design

Recursion Tree Method