Outline

Complexity

Basic tools

Big-Oh

Big Omega

Big Theta

Examples

Lecture 3: Analysing Complexity of Algorithms

Big Oh, Big Omega, and Big Theta Notation

Georgy Gimelfarb
COMPSCI 220 Algorithms and Data Structures

1 / 15

Outline

Complexity

Basic tools

Big-Oh

Big Omega

Big Theta

Examples

Complexity Basic tools Big-Oh Big Omega Big Theta Examples

2 / 15

Outline

Complexity

Basic tools

Big-Oh

Big Omega

Big Theta

Examples

Typical Complexity Curves

Running time is proportional to: T (n) log n T (n) n T (n) n log n T (n) n2 T (n) n3 T (n) nk T (n) 2n T (n) k n ; k > 1

T (n) Complexity: logarithmic linear linearithmic quadratic cubic polynomial exponential exponential

3 / 15

Outline

Complexity

Basic tools

Big-Oh

Big Omega

Big Theta

Examples

Separating an Algorithm Itself from Its Implementation

Two concepts to separate an algorithm from implementation: The input data size n, or the number of individual data items in a single data instance to be processed.
The number of elementary operations f (n) taken by an

algorithm, or its running time. The running time of a program implementation: cf (n)
The constant factor c can rarely be determined and depends

on a computer, operating system, language, compiler, etc. When the input size increases from n = n1 to n = n2 , all other factors being equal, the relative running time of the program (n2 ) cf (n1 ) f (n1 ) increases by a factor of T T (n1 ) = cf (n2 ) = f (n2 ) .
4 / 15

Outline

Complexity

Basic tools

Big-Oh

Big Omega

Big Theta

Examples

Relative Growth g (n) =

f (n) f (5)

of Running Time

The approximate running time for large input sizes gives enough information to distinguish between a good and a bad algorithm. 52 = 25 1 2 5 10 52 = 25 53 = 125 220 106 Input size n 53 = 125 1 3 25 75 54 = 625 56 = 15, 625 2120 1036 54 = 625 1 4 125 500 56 = 15, 625 59 = 1, 953, 125 2620 10187
5 / 15

Function f (n) 5 Constant 1 1 Q Logarithmic log5 n 1 Q Linear n 1 Q Linearithmic n log5 n 1 Q Quadratic n2 1 Q Cubic n3 1 Q Exponential 2n 1

Outline

Complexity

Basic tools

Big-Oh

Big Omega

Big Theta

Examples

Big-Oh, Big-Theta, and Big-Omega Tools

Let f (n) and g (n) be nonnegative-valued functions dened on nonnegative integers n.
Math notation for of the order of . . . or roughly proportional to. . . : Big-Oh (actually: Big-Omicron) O (. . .) g (n) = O (f (n)) Big-Theta (. . .) g (n) = (f (n)) Big-Omega (. . .) g (n) = (f (n))

Big Oh O(. . .) Formal Denition The function g (n) is O(f (n)) (read: g (n) is Big Oh of f (n)) i there exists a positive real constant c and a positive integer n0 such that g (n) cf (n) for all n > n0 .
The notation i abbreviates if and only if.
6 / 15

Outline

Complexity

Basic tools

Big-Oh

Big Omega

Big Theta

Examples

Example 1.8; p.13: g (n) = 100 log10 n is O(n)

g (n) < n if n > 238 or g (n) < 0.3n if n > 1000. By denition, g (n) is O(f (n)), or g (n) = O(f (n)) if a constant c > 0 exists, such that cf (n) grows faster than g (n) for all n > n0 .

To prove that some g (n) is O(f (n))

means to show that for g and f such constants c and n0 exist.

The constants c and n0 are

interdependent.

g (n) is O(f (n)) i the graph of g (n)

is always below or at the graph of cf (n) after n0 .

7 / 15

Outline

Complexity

Basic tools

Big-Oh

Big Omega

Big Theta

Examples

Big-Oh O(. . .): Informal Meaning

O(f (n)) generalises an asymptotic upper bound. If g (n) is O (f (n)), an algorithm with running time g (n) runs asymptotically, i.e. for large n, at most as fast, to within a constant factor, as an algorithm with running time f (n).
In other words, g (n) for large n may approach cf (n) closer and

closer while the relationship g (n) cf (n) holds for n > n0 .

The scaling factor c and the threshold n0 are interdependent

and dier for dierent particular functions g (n) in O(f (n)). Notations g (n) = O (f (n)) or g (n) is O (f (n)) mean actually

g (n) O(f (n)).

The notation g (n) O (f (n)) indicates that g (n) is a member

of the set O(f (n)) of functions.

All the functions in the set O (f (n)) are increasing with the

same or the lesser rate as f (n) when n .

8 / 15

Outline

Complexity

Basic tools

Big-Oh

Big Omega

Big Theta

Examples

Big-Oh O(. . .): Informal Meaning

O(f (n)) generalises an asymptotic upper bound. If g (n) is O (f (n)), an algorithm with running time g (n) runs asymptotically, i.e. for large n, at most as fast, to within a constant factor, as an algorithm with running time f (n).
In other words, g (n) for large n may approach cf (n) closer and

closer while the relationship g (n) cf (n) holds for n > n0 .

The scaling factor c and the threshold n0 are interdependent

and dier for dierent particular functions g (n) in O(f (n)). Notations g (n) = O (f (n)) or g (n) is O (f (n)) mean actually

g (n) O(f (n)).

The notation g (n) O (f (n)) indicates that g (n) is a member

of the set O(f (n)) of functions.

All the functions in the set O (f (n)) are increasing with the

same or the lesser rate as f (n) when n .

8 / 15

Outline

Complexity

Basic tools

Big-Oh

Big Omega

Big Theta

Examples

Big Omega (. . .)
The function g (n) is (f (n)) i there exists a positive real constant c and a positive integer n0 such that g (n) cf (n) for all n > n0 .
(. . .) is complementary to O (. . .). It generalises the concept of lower bound () in the same

way as O(. . .) generalises the concept of upper bound (): if g (n) is (f (n)) then f (n) is O(g (n)).
Example 1: 5n2 is (n) because 5n2 5n for n 1. Example 2: 0.01n is (log n) because 0.01n 0.5 log10 n for

n 100.

9 / 15

Outline

Complexity

Basic tools

Big-Oh

Big Omega

Big Theta

Examples

Big Omega (. . .)
The function g (n) is (f (n)) i there exists a positive real constant c and a positive integer n0 such that g (n) cf (n) for all n > n0 .
(. . .) is complementary to O (. . .). It generalises the concept of lower bound () in the same

way as O(. . .) generalises the concept of upper bound (): if g (n) is (f (n)) then f (n) is O(g (n)).
Example 1: 5n2 is (n) because 5n2 5n for n 1. Example 2: 0.01n is (log n) because 0.01n 0.5 log10 n for

n 100.

9 / 15

Outline

Complexity

Basic tools

Big-Oh

Big Omega

Big Theta

Examples

Big Omega (. . .)
The function g (n) is (f (n)) i there exists a positive real constant c and a positive integer n0 such that g (n) cf (n) for all n > n0 .
(. . .) is complementary to O (. . .). It generalises the concept of lower bound () in the same

way as O(. . .) generalises the concept of upper bound (): if g (n) is (f (n)) then f (n) is O(g (n)).
Example 1: 5n2 is (n) because 5n2 5n for n 1. Example 2: 0.01n is (log n) because 0.01n 0.5 log10 n for

n 100.

9 / 15

Outline

Complexity

Basic tools

Big-Oh

Big Omega

Big Theta

Examples

Big Theta (. . .)
The function g (n) is (f (n)) i there exists two positive real constants c1 and c2 and a positive integer n0 such that c1 f (n) g (n) c2 f (n) for all n > n0 .
If g (n) is (f (n)) then g (n) is O (f (n)) and f (n) is O (g (n))

or, what is the same, g (n) is O(f (n)) and g (n) is (f (n)):
g (n) is O (f (n)) g (n) c f (n) for n > n . g (n) is (f (n)) f (n) c g (n) for n > n . 1 g (n) is (f (n)) c = c ; c = c2 , and n0 = max{n , n }. 1

Informally, if g (n) is (f (n)) then both the functions have

the same rate of increase.

Example: the same rate of increase for g (n) = n + 5n0.5 and

f (n) = n because n n + 5n0.5 6n for n > 1.

10 / 15

Outline

Complexity

Basic tools

Big-Oh

Big Omega

Big Theta

Examples

Big Theta (. . .)
The function g (n) is (f (n)) i there exists two positive real constants c1 and c2 and a positive integer n0 such that c1 f (n) g (n) c2 f (n) for all n > n0 .
If g (n) is (f (n)) then g (n) is O (f (n)) and f (n) is O (g (n))

or, what is the same, g (n) is O(f (n)) and g (n) is (f (n)):
g (n) is O (f (n)) g (n) c f (n) for n > n . g (n) is (f (n)) f (n) c g (n) for n > n . 1 g (n) is (f (n)) c = c ; c = c2 , and n0 = max{n , n }. 1

Informally, if g (n) is (f (n)) then both the functions have

the same rate of increase.

Example: the same rate of increase for g (n) = n + 5n0.5 and

f (n) = n because n n + 5n0.5 6n for n > 1.

10 / 15

Outline

Complexity

Basic tools

Big-Oh

Big Omega

Big Theta

Examples

Big Theta (. . .)
The function g (n) is (f (n)) i there exists two positive real constants c1 and c2 and a positive integer n0 such that c1 f (n) g (n) c2 f (n) for all n > n0 .
If g (n) is (f (n)) then g (n) is O (f (n)) and f (n) is O (g (n))

or, what is the same, g (n) is O(f (n)) and g (n) is (f (n)):
g (n) is O (f (n)) g (n) c f (n) for n > n . g (n) is (f (n)) f (n) c g (n) for n > n . 1 g (n) is (f (n)) c = c ; c = c2 , and n0 = max{n , n }. 1

Informally, if g (n) is (f (n)) then both the functions have

the same rate of increase.

Example: the same rate of increase for g (n) = n + 5n0.5 and

f (n) = n because n n + 5n0.5 6n for n > 1.

10 / 15

Outline

Complexity

Basic tools

Big-Oh

Big Omega

Big Theta

Examples

Comparisons: Two Crucial Ideas

The exact running time function g (n) is not very important

since it can be multiplied by an arbitrary positive constant, c.

The relative behaviour of two functions is compared only

asymptotically, for large n, but not near the origin where it may make no sense.
If the constants c involved are very large, the asymptotical

behaviour loses practical interest!

In most cases, however, the constants remain fairly small. To prove that g (n) is not O (f (n)), (f (n)), or (f (n)), one

has to show that the desired constants do not exist, i.e. lead to a contradiction. g (n) and f (n) in the Big-Oh, -Omega, and -Theta denitions mostly relate, respectively, to exact and rough approximate (like log n, n, n2 , etc) running time on inputs of size n.
11 / 15

Outline

Complexity

Basic tools

Big-Oh

Big Omega

Big Theta

Examples

Example 1.11, p.14

Prove that linear function g (n) = an + b; a > 0, is O(n). The proof: By the following chain of inequalities: g (n) an + |b| (a + |b|)n for all n 1 Do not write O(2n) or O(an + b) as this means still O(n)! O(n)-time: T (n) = 108 + n T (n) = 3n + 1 8 T (n) = 50 + 10 n T (n) = 106 n + 1

Remember that Big-Oh, as well as Big-Omega and

Big-Theta, describes an asymptotic behaviour for large problem sizes. Only the dominant terms as n need to be shown as the argument of Big-Oh, Big-Omega, and Big-Theta.
12 / 15

Outline

Complexity

Basic tools

Big-Oh

Big Omega

Big Theta

Examples

Example 1.12, p.15

The polynomial Pk (n) = ak nk + ak1 nk1 + . . . + a2 n2 + a1 n + a0 ; ak > 0, is O(nk ). The proof: Pk (n) (ak + |ak1 | + . . . + |a0 |)nk for n 1.
Do not write O (Pk (n)) as this means still O (nk )!
k O (n )-time:

Is it also O(n3 )? T (n) = 3n2 + 5n + 1 is O(n2 ) 8 3 8 2 3 T (n) = 10 n + 10 n + 30 is O(n ) Is it also (n2 )? T (n) = 108 n8 + 1000n + 1 is O(n8 ) Is it also (n8 )? O(nm ); m k (nk ); T (n) = Pk (n) is (nm ); m k
13 / 15

Outline

Complexity

Basic tools

Big-Oh

Big Omega

Big Theta

Examples

Example 1.12, p.15

The polynomial Pk (n) = ak nk + ak1 nk1 + . . . + a2 n2 + a1 n + a0 ; ak > 0, is O(nk ). The proof: Pk (n) (ak + |ak1 | + . . . + |a0 |)nk for n 1.
Do not write O (Pk (n)) as this means still O (nk )!
k O (n )-time:

Is it also O(n3 )? T (n) = 3n2 + 5n + 1 is O(n2 ) 8 3 8 2 3 T (n) = 10 n + 10 n + 30 is O(n ) Is it also (n2 )? T (n) = 108 n8 + 1000n + 1 is O(n8 ) Is it also (n8 )? O(nm ); m k (nk ); T (n) = Pk (n) is (nm ); m k
13 / 15

Outline

Complexity

Basic tools

Big-Oh

Big Omega

Big Theta

Examples

Example 1.13, p.15

The exponential function g (n) = 2n+k , where k is a constant,

is O(2n ) because 2n+k = 2k 2n for all n.

Generally, g (n) = mn+k is O (ln ); l m > 1, because

mn+k ln+k = lk ln for any constant k . A brute-force search for the best combination of n binary decisions by exhausting all the 2n possible combinations has exponential time complexity!
230 109 = 1, 000, 000, 000 and

240 1012 = 1, 000, 000, 000, 000

Therefore, try to nd a more ecient way of solving the

decision problem if n 30 . . . 40.

14 / 15

Outline

Complexity

Basic tools

Big-Oh

Big Omega

Big Theta

Examples

Example 1.14, p.15

For each m > 1, the logarithmic function g (n) = logm (n) has the same rate of increase as lg(n), i.e. log2 n, because logm (n) = logm (2) lg(n) for all n > 0.
Omit the logarithm base when using Big-Oh, Big-Omega, and Big-Theta notation: logm n is O(log n), (log n), and (log n).

You will nd later that the most ecient search for data in an ordered array has logarithmic time complexity.

15 / 15