CS306: DAA, 2012, IIITDM Jabalpur

Algorithmic Complexity - I
Atul Gupta

Algorithm
An algorithm is a sequence of computational steps that transform the input into the output An Example: Sorting Problem Input: A sequence of n numbers a1 , a2 , ..., an Output: Output: A permutation (reordering) {a1 , a2 , ..., an} of the input sequence such that a1 <= a2 <= , ..., <=an

CS306: DAA, 2012, IIITDM Jabalpur

Algorithms as a tool
What kinds of problems are solved by Algorithms?
Internet Operating Systems Machine Learning Design and Manufacturing E-Commerce Workflows

Algorithm
Can be expressed in many kind of notations, like
Natural languages, Pseudocode, Flowcharts, Programming Languages

CS306: DAA, 2012, IIITDM Jabalpur

Search
Example: Searching Problem Input: A sequence of n numbers a1 , a2 , ..., an and an element x to be search Output: Output: return the index of the element x if present, else, return not found 4 x 2 9 5 3

Sequential Search Algorithm and C Code

int sequentialSearch(x, A)
int i; for (i=0; i < length[A] and x <> A[i]; i++); if (i == length[A]) return (-1) int sequentialSearch(int c[ ], int s_element){ else return (i)

Algorithm

int i; int array_ size = size_of_array(c); for (i=0; I < array_size and s_element != c[i]; i++); if (i == array_size) return -1; else return i; C-code

CS306: DAA, 2012, IIITDM Jabalpur

Sequential Search
int sequentialSearch(x, A) int i; for (i=0; i < length[A]; i++) { if x == A[i] then break; } if (i == length[A]) return (-1) else return (i) 1 1 1 length[A] 0 - length[A] 1 1 1 1 1

Best Case : when x is the first element (No of comparisons = 1) : O(1) Worst Case : when x is absent (No of comparisons = n) : O(n) Average Case :

1 n (n + 1) : = O(n) i= n i =1 2

Binary Search
BinarySearch(x, A) left = 0; right = length(A) 1; while (left <= right) { mid = (left + right) / 2 if (A[mid] == x) return mid; else if (A[mid] > value) right = mid - 1 else left = mid + 1 } return -1 // not found

1 left

5 mid

9 right

Best Case : when x is mid element (No of comparisons = 1) : O(1) Worst Case : when x is absent (No of comparisons = n) : O(log n) Average Case : O(log n)

CS306: DAA, 2012, IIITDM Jabalpur

Finding Minimum of a Collection

Time Space n 1+1 n 1 1

int findMin(int[] x) { int k = 0; int n = x.length; for (int i = 1; i < n; i++) { if (x[i] < x[k]) { k = i; } } return k; }

1 n+3 O(n) n+2 O(n)

How good is your program?

Sequence of computational steps Running Time
Types of operations H/W Compiler, OS Inputs
scanf(%d, &x); if (x < 10) x++; else for (i =0, i<10, i++);

CS306: DAA, 2012, IIITDM Jabalpur

How good is your program?

Evaluation of Goodness should be independent of
Language Processor (Hardware)

Efficient
Speed (Time) Space (Memory)

How good is your program?

Running time as a function of input size
Input type and size is known Compute number of times each statement is going to be executed (expressed in terms of input size) Replace operation times with constants
Different operations take different (but constant) time

Compute T (n) = time taken by ith statement

CS306: DAA, 2012, IIITDM Jabalpur

An Example: Insertion Sort

Sequence is 4, 2, 9, 7, 8, 3, 5
4 2 4 3 4 7 9 7 8 9 8 9 9 9

The figure is taken from the Book by Cormen et. al. Introduction to Algorithms

Algorithm: Insertion sort

void insertion-sort(A)
1 for (i 2; i < length[A]; i++) { 2 key A[i];
//Insert A[j] into the sorted sequence A[1 ] to A[j - 1].

3 4 5 6 7 }

j i 1; while (j > 0 and A[j] > key) { A[j + 1] A[j]; j j 1; } A[j + 1] key;

CS306: DAA, 2012, IIITDM Jabalpur

Insertion sort: Time Taken

S. No.

Steps
for (i key 2; i< length[A]; i++) A[i];

cost c1 c2 c3
A[j];

times n-1 n-1 n-1 j=2n tj j=2n (tj-1) j=2n (tj-1) n-1

1 2 3 4 5 6 7

//Insert A[j] into the sorted sequence A[1 ] to A[j - 1].

i-1

while (j > 0 and A[j] > key) A[j + 1] j A[j + 1] j 1; key

c4 c5 c6 c7

T(n) = c1(n-1)+ c2(n-1)+ c3(n-1) + c4 j=2n tj + c5 j=2n(tj-1) + c6 j=2n(tj-1) + c7(n-1)

Insertion Sort: Time Taken

Best Case Worse Case Average Case

CS306: DAA, 2012, IIITDM Jabalpur

Why Worse Case?

Act as the Upper Bound Worse case do occur in practice
Ex: a record not found in the database

Average case is often (roughly) as bad as the worse case

Algorithmic Complexity
A measure of the performance of an algorithm with respect to input size
Space complexity Time complexity (Mostly used)

Expressed in terms of Asymptotic Notations

Bog-Oh (O), Big-Theta (), Big-Omega ()

What to measure
Worst case performance Average case Best case

CS306: DAA, 2012, IIITDM Jabalpur

Big-Oh (O)
An indicator of the worst case performance (upper bound) Defined as a function T(n) which is said to be of O(g(n)) if there exist positive constants c0 and n0 such that for all n >= n0, we have T(n) < = c0 g(n) T(n) is asymptotically smaller than or equal to g(n)
cg(n) T(n)

n0

T(n) <= c0 g(n)

Big-Oh (O): Examples

T(n) = 3n2+4n = O(n2) T(n) = 3n2+4n = O(n3) T(n) = 3n2+4n O(n) T(n) = (n+1)2 = O (n2)

10

CS306: DAA, 2012, IIITDM Jabalpur

Big-Omega ()
An indicator of the best case performance (lower bound) Defined as a function T(n) which is said to be of (g(n)) if there exist positive constants c0 and n0 such that for all n >= n0, we have T(n) > = c0 g(n) T(n) is asymptotically greater than or equal to g(n)
T(n) c0g(n)

n0

T(n) >= c0 g(n)

Big-Omega (): Examples

T(n) = 3n2+4n = (n2) T(n) = 3n2+4n = (n) Tn = 4n+2 = (n) Tn = 4n+2 = (1)

11

CS306: DAA, 2012, IIITDM Jabalpur

Big-Theta ()
An indicator of the worse case and best case performance (asymptotically tight bound) Defined as a function T(n) which is said to be of (g(n)) if there exist positive constants c1, c2 and n0 such that for all n >= n0, we have c1 g(n) <= T(n) <= c2 g(n) T(n) is asymptotically equal to g(n)
c2g(n) T(n) c1g(n)

n0

T(n) <= c0 g(n)

Big-Theta (): Examples

T(n) = 3n2+4n = (n2) T(n) = 3n2+4n (n) Tn = 4n+2 = (n) Tn = 4n+2 (1)

12

CS306: DAA, 2012, IIITDM Jabalpur

Order of Growth
Analysis Type Big O Big Big Mathematical Expression T(N) = O( F(N) ) T(N) = ( F(N) ) T(N) = ( F(N) ) Relative Rates of Growth T(N) < F(N) T(N) > F(N) T(N) = F(N)

1 < logn < n < n log n < n2 < n3 < 2n <!n

Name Convention
O(1) O(logc n) O(logc n) O(n) O(nc) O(cn), c>1 O(n!) Constant Logarithmic (same order c) Polylogarithmic Linear Polynomial Exponential Factorial

13

CS306: DAA, 2012, IIITDM Jabalpur

Complexity and Tractability

T ( n)
n 10 20 30 40 50 100 103 104 105 106 n n log n n .01s .03s .1s .02s .09s .4s .03s .15s .9s .04s .21s 1.6s .05s .28s 2.5s .1s .66s 10s 1ms 1s 9.96s 10s 130s 100ms 10s 100s 1.66ms 1ms 19.92ms 16.67m
2

n3 n4 1 s 10s 8 s 160s 27s 810s 2.56ms 64s 6.25ms 125s 1ms 100ms 1s 16.67m 16.67m 115.7d 11.57d 3171y 31.71y 3.17107y

2n 1s 1 s 2.84h 1ms 6.83d 1s 121d 18m 3.1y 13d 3171y 41013y 3.171013y 3210283y 3.171023y 3.171033y 3.171043y

n10

Assume the computer does 1 billion ops per sec.

27

log n 0 1 2 3 4 5

n 1 2 4 8 16 32

n log n 0 2 8 24 64 160

n2 1 4 16 64 256 1,024

n3 1 8 64 512 4096 32,768

2n 2 4 16 256 65,536 4,294,967,296

70000 60000 50000 40000 30000

2n

n2

100000

2n

n3

n2 n log n n

10000

n3

1000

100

20000 10000 0 n

n log n n log n

10

log n
n

28

14

CS306: DAA, 2012, IIITDM Jabalpur

Time Complexity How much it matters?

One million number

Computer - A 109 instructions/sec Insertion sort (2n2)

Computer - B 107 instructions/sec Merge sort 50 n log n

2000 Seconds

100 Seconds

Insertion sort vs. Merge sort

Insertion sort (2n2) Merge sort (32 n logn)

15

CS306: DAA, 2012, IIITDM Jabalpur

Insertion: Array or Linked List?

Describe how a new integer could be inserted into a sequence of sorted integers (so that the resulting sequence remains sorted) in each of the following cases. (i) The sequence is stored in an array. (ii) The sequence is stored in a linked list. (iii) Compare the efficiency of (i) and (ii).

Summary
Algorithms are omnipresent in computational domain Time complexity of algorithm does matter Time complexity of an algorithm is measured as rate of growth of the running time wrt input size Time complexity is normally expressed
Big-Oh (O) ----- upper bound (most Popular) Big-theta () ----- tight (upper & lower) bound Big-omega () ----- lower bound

Can you measure time complexity of a given algorithm?

16