Big-O Big-O notation is one of the ways in which we talk about how complex an algorith m or program is.

It gives us a nice way of quantifying or classifying how fast o r slow a program is as a function of the size of its input, and independently of the machine on which it runs. Examples Let's look at a program (in linear1.cpp): #include <cstdio> #include <cstdlib> #include <iostream> using namespace std; main(int argc, char **argv) { int n; double f; double count; int i, j; long t0; if (argc != 2) { fprintf(stderr, "Usage: linear1 n\n"); exit(1); } n = atoi(argv[1]); t0 = time(0); count = 0; f = 23.4; for (i = 0; i < n; i++) { count++; f = f * f + 1; j = (int) f / 1000; f = f - j*1000; } printf("N: %d Count: %.0lf } What this does is compute n random-ish numbers between one and 1000, and if we u ncommented the printf() statement, then it would print them -- try it out (uncom ment the print statement). Suppose we run this program with varying values of n. What do we expect? Well, a s n increases, so will the count, and so will the running time of the program: (This is on my machine at home -- I don't know how fast the machines here will b e.) UNIX> g++ -o linear1 linear1.c UNIX> linear1 1 N: 1 Count: 1 Time: 0 UNIX> linear1 10000 N: 10000 Count: 10000 Time: 0 UNIX> linear1 10000000 N: 10000000 Count: 10000000 Time: 8 UNIX> linear1 20000000 N: 20000000 Count: 20000000 Time: 14

Time: %ld\n", n, count, time(0)-t0);

UNIX> linear1 100000000 N: 100000000 Count: 100000000 Time: 72 UNIX> Just what you'd think. The running time is roughly linear: running time = 72n/100000000 Obviously, count equals n. Now, look at four other programs below. I will just show their loops: linear2.cpp: ... for (i = 0; i < n; i++) { count++; f = f * f + 1; j = (int) f / 1000; f = f - j*1000; } for (i = 0; i < n; i++) { count++; f = f * f + 1; j = (int) f / 1000; f = f - j*1000; } ... log.cpp: ... for (i = 1; i <= n; i *= 2) { count++; f = f * f + 1; j = (int) f / 1000; f = f - j*1000; } ... nlogn.cpp: ... for (k = 0; k < n; k++) { for (i = 1; i <= n; i *= 2) { count++; f = f * f + 1; j = (int) f / 1000; f = f - j*1000; } } ... nsquared.cpp: ... for (i = 0; i < n; i++) { for (k = 0; k < n; k++) { count++; f = f * f + 1; j = (int) f / 1000; f = f - j*1000; } } ... In the five programs total, the value of count will be the following (note, I wi ll expect you to be able to do things like tell me what count is as a function o f n on tests):

linear1.cpp: count = n. linear2.cpp: count = 2n. log.cpp: count = log(n). (where the logarithm is base 2). nlogn.cpp: count = n*log(n). (where the logarithm is base 2). nsquared.cpp: count = n*n. In each program, the running time is going to be directly proportional to count. Why? Read chapter two for how to count instructions. So, what do the running ti mes look like if you increase n to large values. I have the output of the variou s programs in the following table of links: linear1 linear2 log nlogn nsquared Some things you should notice right off the bat: log(n) is very, very small in c omparison to n. This means that log.cpp is blazingly fast for even huge values o f n. On the other end of the spectrum, n*n grows very quickly as n increases. Be low, I graph all of the programs' running times as a function of n: So, this shows what you'd think: log(n) < n < 2n < n*log(n) < n*n Perhaps its hard to gauge how much each is less than the other until you see it. Below I plot the same graph, but zoomed up a bit so you can get a better feel f or n*log(n) and n*n.

Back to Big-O: Function comparison Big-O notation tries to work on classifying functions. The functions that we car e about are the running times of programs. The first concept when we deal with B ig-O is comparing functions. Basically, we will say that one function f(n)is gre ater than another g(n) if there is a value x so that for all i >= x: f(i) >= g(i) Put graphically, it means that after a certain point on the x axis, as we go rig ht, the curve for f(n) will always be higher than g(n). Thus, given the graphs a bove, you can see that n*n is greater than n*log(n), which is greater than 2n, w hich is greater than n, which is greater than log(n). So, here are some functions: a(n) = 1 b(n) = 100 c(n) = 6-n d(n) = n e(n) = 2n f(n) = 2n-5 g(n) = n*n - 5000000000 h(n) = log(n) i(n) = log(n) - 100 j(n) = n*log(n)-100 So, we can ask ourselves questions: Is b(n) > a(n)? Yes. Why? Because for any va lue of n, b(n) is 100, and a(n) is 1. Therefore for any value of n, b(n) is grea ter than a(n). That was easy. How about c(n) and b(n)? b(n) is greater, because for any value o f n greater than 6, b(n) is 100 and c(n) is negative. Here's a total ordering of the above. Make sure you can prove all of these to yo urselves: g(n) > j(n) > e(n) > f(n) > d(n) > h(n) > i(n) > b(n) > a(n) > c(n) Some rules:

If f(n) and g(n) are both polynomials of degree k, then the lead coefficient def ines which one is greater. If f(n) is a polynomial of degree k and g(n) is a polynomial of degree l < k, an d both lead coefficients are positive, then f(n) > g(n). If a*f(n) > b*g(n) for all positive constants a and b, then c*f(n) + d*g(n) > e* f(n) + x*g(n) if and only if c > e. For example, 20n*n - 100n > 19n*n + 60,000n. Big-O Given the above, we can now define Big-O, Big-Omega and Big-Theta. We start with Big-O: T(N) = O(f(N)) if there exists a constant c such that c*f(N) >= T(N). Given the definitions of a(n) through j(n) above: a(n) = O(1). b(n) = O(1). b(n) = O(n). This is because n > b(n). b(n) = O(n*n). This is because n*n > b(n). b(n) = O(a(n)). a(n) = O(b(n)). e(n) = O(n). This is because 2n >= e(n). I.e. with c = 2, cn >= e(n). i(n) = O(log(n)). j(n) = O(n*log(n)) g(n) = O(n*n) Note that O(f(N)) is an upper bound on T(N). That means that T(N) is definitely not slower than f(n). Similarly, if g(N) > f(N) and T(N) = O(f(N)), then T(N) = O(g(n)) too. The definitions of the others are similar: Big-Omega: T(N) = O(f(N)) if f(N) = O(T(N)). Big-Theta: T(N) = T(f(N)) if T(N) = O(f(N)) and T(N) = O(f(N)). At this point, I think that giving the Wikipedia page on Big-O a scan is a good idea. This includes: The introduction. The first two equations in the Formal Definition. The Example. The Infinite asymptotics section. The Equals Sign section. The Orders of common functions section (ignore the "L-notation" line). The definitions of Big-O, Big-Omega and Big-Theta in the Family of Bachmann-Land au notations section. The text starting with "Aside from Big-O notation, ..." until the end of the sec tion. Two Big-O Proofs Is n*n + n + 1 = O(n*n)? See the following PDF file for a proof. Generalizing, is an*n + bn + d = O(n*n) for a,b,d > 1 and b > d? See the followi ng PDF file for a proof.