REFERENCES:

http://www.youtube.com/watch?v=EL9T1ngiCqA http://video.google.com/videoplay?docid=6724701313234177393

Asymptotic Notation
Formal way notation to speak and classify them Asymptotic analysis refers to classifying the behaviour of functions

Two Features:

there are consts c > 0 and n0 > 0 such that 0 <= f(n) <= cg(n) for all n > n0 Example: 2n^2 = O(n^3) Set definition O(g(n)) = { f(n) : there are consts c > 0, n0 > 0 such that 0 <= f(n) <= g(n) for all n > n0 } Example:

Or

Macro Definition A set in a formula represents an anonymous function in that set. Example:

means there is a function Example:

such that

means: for any such that

, there is an

Omega Notation M(g(n)) = { f(n) : there exist consts c > 0, n0 > 0 such that 0 <= c g(n) <= f(n) for all n > n0 } Example:

Analogies: O <= M >= theta =

Theta notation thetha(g(n)) = { f(n) : there exist consts c1 > 0, c2 > 0, n0 > 0 such that c1g(n) <= f(n) <= c2g(n) for all n >= n0 Example:

o & m notations (< / >) inequality must hold for all consts c > 0 Example:

Solving recurrences Substitution method 1. Guess the form of the solution 2. Verify by induction 3. Solve for consts Example: T(n) = 4T(n/2) + n T(1) = O(1) Guess: T(n) = O(n^3) Assume: T(k) <= c k^3 for k < n Thus, T(n) = 4T(n/2) + n T(n) <= 4 c(n/2)^3 + n <= 1/2cn3 + n = cn^3 residual <= cn^3 , if 1/2cn^3 n >= 0 Try: T(n) = O(n^2) Assume: T(k) <= ck^2 for k < n T(n) = 4T(n/2) + n <= 4c(n/2)^2 + n = cn^2 + n = cn^2 (n ) Want: n >= 0 Assume: T(k) <= c1k^2 c2k T(n) = 4T(n/2) + n

<= 4(c1(n/2)^2 c2(n/2)) + n = c1n^2 2c2n + n = c1n^2 c2n (n + c2n) Want: n + c2n >= 0 Recursion-Tree Method Example: T(n) = T(n/4) + T(n/2) + n^2 Expand:

< n leaves Total (level-by-level) n^2 5/16 n^2 25/256 n^2

5^kn^2

= (1 + 5/16 + 25/256 + )n^2 RECALL: 1 + + + 1/8 + 1/64+ =1+1=2 Thus, <= 2n^2 = O(n^2) Master Method applies to recurrences of the form T(n) = aT(n/b) + f(n) Where: a>= 1, b > 1,

f(n) is asymptotically positive f(n) > 0, for n >= n0

Compare:

f(n) with n^(lg ba)

MITs Introduction to Algorithms, Lectures 1 and 2: Analysis of Algorithms

I just finished watching the last lecture of MITs Introduction to Algorithms course. Having a great passion for all aspects of computing, I decided to share everything I learned with you, my dear readers! This is the first post in an article series about this course. As I wrote earlier, I am very serious about watching video lectures. If they are math-intensive, I usually take notes as if I were in the classroom. Lectures in this course were exactly like that logarithms, big-os, thetas, expectations, and all the other math guys fighting with each other on the blackboards. There are totally 23 video lectures, each around 1 hour 20 minutes long. I will be posting about 2 3 lectures at a time which will result in approximately 10 blog posts. Each post will contain annotated lecture, along with embedded video of the lecture and a time-stamped list of topics covered in the lecture. I will also post the notes I took myself as I watched the lectures (actually, I just bought a scanner (Canon CanonScan 4400F) just for this purpose!) Understanding and designing effective algorithms is a very important skill for a top-notch programmer. You can still do good without knowing much about algorithms, but knowing them makes you superior. There are two kinds of people, those who can design effective algorithms and those who dont. Lets start with Lecture 1 of this course.

Lecture 1: Analysis of Algorithms

The first lecture is given by the famous professor Charles E. Leiserson. He is the L in CLRS. If that doesnt ring you a bell - its one of the most popular books on algorithms! He starts the lecture by explaining what this course and algorithms will be all about. He says that this course will be about Analysis of Algorithms and states: Analysis of algorithms is the theoretical study of computer program performance and resource usage. Designing great software is not just about performance. Charles presents a list of 12 things that can be more important than performance. Just for comparison with algorithms guru, what do you think can be more important than performance? Here is the list of things more important than performance that Charles presented:

modularity, correctness, maintainability, security, functionality, robustness, user-friendliness, programmers time, simplicity, extensibility, reliability, and scalability.

He also asks Why study algorithms and performance at all?. He and students answer:

Sometimes performance is correlated with user-friendliness. Performance draws line between feasible and unfeasible. Algorithms give language for talking about program behavior. Performance can be used to pay for other things, such as security, features and userfriendliness.

The lecture continues with the definition of the Sorting Problem - given a sequence (a1, a2, , an) of numbers, permute them in such a way that a1 <= a2 <= ... <= an. There are various algorithms to solve this problem. Two algorithms are presented to solve this problems - one of them is Insertion Sort and the other is Merge Sort. Running time of these algorithms is analyzed by introducing Asymptotic Analysis and Recursion Trees. Here is the whole lecture: Direct URL: http://video.google.com/videoplay?docid=-2333306016564732003 Topics covered in lecture 1:

[17:15] Main topic of the course - Analysis of algorithms. [19:00] Whats more important than performance? [22:03] Why study algorithms and performance? [27:45] The sorting problem. [29:30] Insertion sort algorithm [34:30] Example of Insertion sort. [36:25] Running time of algorithms. [39:39] Definition of worst-case, average-case and best-case types of analysis. [46:50] How to analyze the Insertion sorts worst-case running time?

[49:28] BIG IDEA - Asymptotic analysis. [50:49] Asymptotic notation - theta notation. [57:14] Insertion sort analysis. [01:02:42] Is Insertion sort fast? [01:03:40] Merge sort algorithm. [01:05:25] Example of Merge subroutine of Merge sort. [01:08:15] Analysis of Merge sorts running time. [01:10:55] Recurrence equation for Merge sort. [01:13:15] Recursion tree solution of the Merge sorts recurrence equation.

Lecture 1 notes:

Lecture 1, page 1 of 2.

Lecture 1, page 2 of 2.

Lecture 2: Asymptotic Notation

Lecture 2, on the other hand, is given by genius professor Erik Demaine. He is the youngest professor in the history of the MIT! He became professor at MIT at 20! Wow! This lecture is all about mathematical notation (Asymptotic Notation) used in the analysis of algorithms. Its the big-o notation, big omega notation, theta notation, small-o and small-omega notation. The second half of the lecture is devoted to solving recurrence equations. Three methods are presented:

Substitution method,

Recursion-tree method, and The Master method.

Here is the whole lecture: Direct URL: http://video.google.com/videoplay?docid=6724701313234177393 Topics covered in lecture 2:

[01:25] Big-o (upper bounds) notation. [03:58] Set definition of big-o notation. [05:25] The meaning of O(h(n)) in notation f(n) = g(n) + O(h(n)). [10:20] Big-omega (lower bounds) notation. [11:40] Analogies of O, and to comparison operations of real numbers. [12:28] Theta (tight bounds) notation. [13:40] Small-o and small-omega notation. [17:03] Solving recurrences: substitution method. [37:56] Recursion-tree method. [49:00] The Master method. [01:02:00] Proof sketch of the Master method.

Lecture 2 notes:

Lecture 2, page 1 of 2.

Lecture 2, page 2 of 2.

Lecture 2. Sketch of Masters theorem proof.

Have fun absorbing all this information! Until next post! Ps. It turned out that the lectures were not available anywhere but from MITs OCW website. I found that they were released under CC license, which allowed me to upload them to Google Video, so I can embed them in the posts! Pps. the lectures are taught from the CLRS book (also called Introduction to Algorithms):