Lecture 1 Introduction

What is an Algorithm?
An algorithm is a computational procedure that takes some value, or a set of values, as input and produces some values, or a set of values, as output. So, it is a sequence of computational steps that transform the input into the output. Correction: sequence -> combination

Sequential and Parallel

In sequential computation, an algorithm is a sequence of computational steps. However, it is not true in parallel computation. In this course, we study only algorithms in sequential computation.

Algorithms for Sorting

Input : A sequence of n numbers {a1 , a2 ,..., an }. Output : A permutation {a '1 , a '2 ,..., a 'n } of input sequence such that a '1 e a '2 e . e a 'n .
e.g.,

Input : 5, 2, 4, 6, 1, 3. Output :1, 2, 3, 4, 5, 6.

Insertion Sort, Merge Sort

Efficiency
Running time from receiving the input to producing the output.
Running time

Insert Sort Merge Sort

O(n ) O(n log n)

f (n) ! O( g (n)) means there exist constants c " 0 and n0 " 0 such that 0 e f (n) e cg (n) for n u n0 .

c1n

and c2 n log n, who is bigger?

When n is sufficiently large, c1n 2 " c2 n log n. Therefore, what we really care about is the growth of the function!

Insertion Sort

input array A
for j n 2 to length[ A] do key n A[ j ] i n j 1 while i " 0 and A[i ] " key do A[i  1] n A[i ] i n i 1 A[i  1] n key

for j n 2 to length[ A] do begin key n A[ j ]; i n j  1; while i " 0 and A[i ] " key do begin A[i  1] n A[i ]; i n i  1; end - while; A[i  1] n key; end - for.

U 5, 2, 4, 6, 1, 3 U 2, 5, 4, 6, 1, 3 U 2, 5, 4, 6, 1, 3 U 2, 4, 5, 6, 1, 3 U 2, 4, 5, 6, 1, 3

key

U 2, 4, 5, 6, 1 , 3 U 2, 4, 5, 1 , 6, 3 U 2, 4, 1 , 5, 6, 3 Y 2, 1 , 4, 5, 6, 3 U 1 , 2, 4, 5, 6, 3

U 1, 2, 4, 5, 6, 3 U 1, 2, 4, 5, 3, 6 U 1, 2, 4, 3, 5, 6 U 1, 2, 3, 4, 5, 6

How to calculate running time?

Each line of pseudocode requires a constant time. (In RAM model)

for j n 2 to length[ A] do begin key n A[ j ]; i n j  1;

This loop runs n-1 times and each time runs at most 4+3(j-1) lines.

while i " 0 and A[i ] " key This loop runs at most do begin A[i  1] n A[i ]; i n i  1; end - while; A[i  1] n key; end - for.
T (n) e j ! 2 (4  3( j  1)) (n  1)(n  2) ! n 1 3 2
n

j-1 times and each time runs at most 3 lines.

Remark on Running Time

Running time is a function of input size. In Turing machine model, the input size of sorting is

log 2 a1  log 2 a2  .  log 2 an Hence, " A[i ] " key" runs not in constant time.

Divide and Conquer

Divide the problem into subproblems. Conquer the subproblems by solving them recursively. Combine the solutions to subproblems into the solution for original problem.

Merge Sort
Let Merge - Sort ( A, p, r ) be a procedure which sorts elements in subarray A[ p...r ]. Main Program begin Merge - Sort ( A,1, n); end

Procedure
Merge - Sort ( A, p, r ) if p r Merge - Sort ( A, p, q ); Merge - Sort ( A, q  1, r ); Merge( A, p, q, r ); end - then. then begin q n ( p  r ) / 2;

Merge( A, p, q, r ) n1 n q  p  1; n2 n r  q ; create array L[1..n1  1] and R[1..n2  1]; for i n 1 to n1 do L[i ] n A[ p  i  1]; for j n 1 to n2 do R[ j ] n A[q  j ];

L(n1  1) n g; R( n2  1) n g; i n 1; j n 1; for k n p to r do if A[i ] e R[ j ] then A[ k ] n L[i ] and i n i  1 else A[ k ] n R[ j ] and j n j  1;

Example

T L : 2, 3, 5, 7, g; A : 1, T L : 2, 3, 5, 7, g; A :1, 2, T L : 2, 3, 5, 7, g; A :1, 2, 3, T L : 2, 3, 5, 7, g; A :1, 2, 3, 4, T L : 2, 3, 5, 7, g; A :1, 2, 3, 4, 5,

T R : 1 , 4, 6, 8, g; T R :1, 4, 6, 8, g; T R :1, 4, 6, 8, g; T R :1, 4, 6, 8, g; T R :1, 4, 6, 8, g;

T T L : 2, 3, 5, 7, g; R :1, 4, 6, 8, g; A : 1, 2, 3, 4, 5, 6, T T L : 2, 3, 5, 7, g; R :1, 4, 6, 8, g; A :1, 2, 3, 4, 5, 6, 7, T T L : 2, 3, 5, 7, g; R :1, 4, 6, 8, g; A :1, 2, 3, 4, 5, 6, 7, 8, T T L : 2, 3, 5, 7, g; R :1, 4, 6, 8, g;

Procedure
Merge - Sort ( A, p, r ) if p r then begin q n ( p  r ) / 2; Merge - Sort ( A, p, q ); Merge - Sort ( A, q  1, r ); Merge( A, p, q, r ); end - then.
T (n)

T (n / 2) T (n / 2)

5(n)

T (n) ! 2T (n / 2)  5(n)

f (n) ! 5( g (n)) means there exist constants c1 u c2 " 0 and n0 " 0 such that 0 e c2 g (n) e f (n) e c1 g (n) for n u n0 .
Symmetry

f (n) ! 5( g (n)) g (n) ! 5( f (n))

How to Solve Recurrences

Substitution method. Recursion-tree method. Master method.

Substitution Method
Guess the form of the solution. Use math induction to find the constants and also show the solution works.

Guess : T ( n) e cn lg n, i.e., T ( n) ! O(n lg n) Note : 2T ( n / 2)  c1n e T ( n) e 2T ( n / 2)  c2 n for some constants c2 u c1 " 0, n u n0 . Induction : (1) For n e n0 , T (n) e T ( n0 ). (2) Assume T ( n / 2) e c(n / 2) lg(n / 2). Then T (n) e 2c(n / 2) lg(n / 2)  c2 n e cn lg n  (c2  c) n. e cn lg n (Take c u max(c2 , T ( n0 )).)

Guess : T (n) e cn lg n, i.e., T (n) ! O(n lg n) Note : 2T ( n / 2)  c1n e T (n) e 2T ( n / 2)  c2 n for some constants c2 u c1 " 0, n u n0 . Induction : (1) For n e n0 , T (n) e T (n0 ). (2) Assume T ( n / 2) e c n / 2lg n / 2. Then T (n) e 2c n / 2lg n / 2  c2 n e 2c (n / 2  1)(lg n  1  2 lg e / n)  c2 n e cn lg n  (c2  0.5c )n for sufficiently large n e cn lg n for sufficiently large n (Take c u max(2c2 , T (n0 )).)

lg n / 2 e lg(n / 2  1) e lg(n / 2)  lg(1  2 / n) e lg n  1  lg e ! lg n  1  (2 lg e) / n

2/ n

(note : 1  x e e )

Remark
If T (n) ! 2T (n / 2)  O(n) then T (n) ! O(n lg n).

Recursion Tree
T (2k ) T (2 k 1 ) T (2 k 1 )

c2 2 k

2c2 2 k 1 ! c2 2 k

T (2 k 2 )

T (2 k  2 )

T (2 k 2 )

T (2 k  2 )

T (1)

T (1)

T (1)

T (1) T ( 2 k ) ! c2 k 2 k

Master Theorem
Let T (n) ! aT (n / b)  f (n) where constants a u 1, b " 1, and n / b means n / b or n / b. If f (n) ! O(n If f (n) ! 5(n
log b a I

) for some constant I " 0, ).

log b a

then T (n) ! 5(n

log b a

log b a

), then T (n) ! 5(n

lg n).

If f (n) ! ;(n ) for some constant I " 0, and if af (n / b) e cf (n) for a constant c 1 and sufficiently large n, then T (n) ! 5( f (n)).

log b a I

f (n) ! ;( g (n)) means there exist constants c " 0 and n0 " 0 such that 0 e cg (n) e f (n) for n u n0 .
Relationship

f (n) ! ;( g (n)) g (n) ! O( f (n))

What we learnt in this lecture?

How to calculate running time. How to solve recurrences. Insertion sort and Merge sort. Divide and conquer Lecture Notes give you key points in each lecture. You must read textbook after lectures in order to study well.