Complexity

The complexity of an algorithm is the amount of a resource, such as
time, that the algorithm requires.  It is a measure of how 'good' the
algorithm is at solving the problem.  The complexity of a problem is
defined as the best algorithm that solves a problem.

We've spent the whole course emphasizing correctness and
understandability, sometimes at the expense of speed.  When
we analyze complexity,
	Better = faster.

But what does faster mean?

Measuring the speed of an algorithm

When you want to compare the speed of two algorithms, you can't just
implement them in programs and time the programs with a stop watch,
because the answer would be muddled by interfering factors:
 * speed of computer
 * choice of programming language
 * choice of compiler and execution environment
 * random events caused by other programs running during 
       execution of the test
 * quality of programming

Instead, we isolate the "important operations" - that is, the ones that
take longest (the ones that are nested the most) - and count them.

We are really interested in the behaviour of an algorithm, as the size
of our data set increases.  We want to know how long the algorithm
takes, in terms of the size of our data set, or some other factor in
our program.

Phone Book Example

Suppose we had a telephone book, b, and we wanted to look up name, x,
and suppose the book has n names.

There are several ways to solve this problem:
Algorithm 1: Linear Search
Algorithm 2: Binary Search
	A heuristic...

Linear Search  (Sequential)

Start with the first name, and continue looking until x is found.
Then print corresponding phone number.

How much time does this take?  We look at the number of comparisons.
	Best case:		1 step
	Average case:		n/2 steps
	Worse case:		n steps

Assume that we have an array, book, of phone book entries (name &
number), and we are searching for the phone number of name x...

	int whereFound = -1;	// position where name found

	// Go through each entry, comparing name to x
	for (int i=0; i<n; i++) {
		// compare next name
		if (book[i].getName().equals(x)) {
			whereFound = i;
			break;
		}
	}

	if (whereFound != -1)
		System.out.println ("Number is " + book[whereFound].getNumber();
	else
		System.out.println ("Name not in phone book.");
	
Is this the best (fastest) algorithm?

Binary Search

Compare x with the middle name in the book.  If x comes before y,
recursively look for x in the first half.  Otherwise recursively look
for x in the second half.  The search is finished when there is only 1
name left in the half you're searching in.

How much time does this take?  Depends on how it's implemented:
	Best case:		log n steps   (could be 1 step)
	Average case:	log n steps
	Worse case:		log n steps

The number of times you go through the loop is proportional to the
number of times you have to divide the list of size n by 2 in order to
reduce it to only one entry … log base 2 of n.

Example

Suppose our book, b, contains these n (8) names, and we are looking for
the name, x, which is "helen."  Suppose that each name has a
corresponding phone number, although the numbers are not shown here.

   begin                  mid                               end
     0       1       2      3       4       5       6        7 
[ anna     carl    doug   earl    fiona  gerard   helen     zack ]
                          begin            mid              end
                                           begin   mid      end
                                                  begin     end

We find "helen" in 3 steps with the binary search, whereas it would have
taken 7 steps with the linear search.   [log n = log 8 = 3]

Assume that we have an array, book, of phone book entries (name &
number), and we are searching for the phone number for the name x...

	// Initialization
	int mid;
	int begin = 0;
	int end = book.length-1;

	// Loop until begin and end are within 1.
	while (begin != end-1) {
		mid = (begin+end)/2;
		if (book[mid].getName().compareTo(x) > 0)
			end = mid;
		else
			begin = mid;
	}

	// Now print out the phone number that is
	// stored in element book[begin] for this name

    if (book[begin].getName().equals(x))
        System.out.println ("Found value   " + book[begin].getNumber());
    // Special case where element is found at last index.
    else if (begin+1 < book.length && book[begin+1].getName().equals(x)) 
        System.out.println ("Found value   " + book[begin+1].getNumber());
    else
        System.out.println ("Value not found.");

Is this the fastest algorithm?

Big-Oh Notation

We use "order of" (or Big-Oh notation) to express the time complexity
of an algorithm.  This in an approximation of the time that it takes to
run.

Suppose that you are comparing two different algorithms that solve a
particular problem.  One has a worse case complexity of n+1
comparisons, and the other has a worse case complexity of n+3.  What
does this mean in terms of Big-Oh?

We want to compare algorithms, as our data set gets really large (as n
increases).  Which of these are faster?

	n + 1     <==>   n + 3       --> both are O(n)
	  n       <==>     2n        --> both are O(n)
	  n       <==>     n^2       --> left is O(n), right is O(n^2)
	n^2 + n   <==>    n^2 + 300  --> both are O(n^2)

We use Big-Oh notation to give a rough estimate of the complexity.
This is usually sufficient.  We remove the additive and multiplicative
factors from the expression, and say that the complexity of the
algorithm is "on the order of" some expression containing n.

If you count the number of operations in an algorithm, and get the
following number of comparisons, what would the complexity be?
	5n comparisons                   --> O(n)
	n + 10 comparisons               --> O(n)
	0.6n + 33 comparisons            --> O(n)
	5 comparisons                    --> O(1)
	log n + 1 comparisons            --> O(log n)
	2n^3 + n + 5 comparisons         --> O(n^3)
	3log n + 2n + 3 comparisons      --> O(n)


Comparing Runtime Functions

	n	  	 log n	n^2		2^n
	1			0	1		2
	128			7	16384	10^38
	256			8	65536	Infinity
	65536		16	10^9	Infinity
	16 Meg		24	10^14	Infinity
	4096 Meg	32	10^19	Infinity

Fastest to Slowest

  O (1) --> O (log n) --> O (n) --> O (n^2) --> O (n^3) --> 
     O (n^4) --> O (2^n) --> O (3^n)


Analyze the complexity

Program 1: O (n^2)

	int sum = 0;
	int num = 35;

	for (int i=1; i<=2*n; i++) {
		for (int j=1; j<=n; j++) {
			num += j*3;
			sum += num;
		}
	}

	for (int k=1; k<=n; k++) {
		sum++;
	}	

Program 2: O (n^3)

	int sum = 0;
	int num = 35;

	for (int i=1; i<=2*n; i++) {
		for (int j=1; j<=n; j++) {
			num += j*3;
			sum += num;
		}
	}

	for (int i=1; i<=n; i++)
		for (int j=1; j<=n; j++)
			for (int k=1; k<=n; k++)
				num += j*3;

Program 3: O (n x m)

	int sum = 0;

	for (int i=1; i<=n; i++) {
		for (int j=1; j<=m; j++) {
			sum++;
		}
	}

Program 4: O (n^2)

	int sum = 0;

	for (int k=1; k<=n; k++) {
		sum++;
	}	

	for (int i=1; i<=n; i++) {
		for (int j=i; j<=n; j++) {
			sum++;
		}
	}

Program 5: O (n)

	int sum = 0;

	for (int i=0; i<n; i++) {
		if (i > n/2)
			sum += 2;
		else
			sum++;
	}

Program 6: O (n^2)

	int sum = 0;

	for (int j=1; j<n; j++) {
		sum++;
	}	

	for (int k=0; k<n; k++) {
		for (int i=0; i<n; i++) {
			sum++;
		}
		for (int p=0; p<n; p++) {
			sum++;
		}
		for (int m=0; m<n; m++) {
			sum++;
		}
	}


Analyze the complexity

1. Find the maximum in an unsorted list of n numbers.
  O (n)

2. Find both the maximum and minimum in an unsorted list.
  O (n)

3. Sort n numbers.
  O (n x log n)

4.  Multiply two n x n matrices
  O (n^3)

5.  Linear Search (sorted/unsorted)
  O (n) - sorted
    O (n) - unsorted

6.  Binary Search (sorted/unsorted)
  O (log n) - sorted
Algo does not work - unsorted

[ CSC108 Home | Outline | Announcements | Newsgroup | Assignments | Lectures | Marks | Links ]

---

© Copyright 2000. All Rights Reserved.