=========================================================================== CSC B63 Lecture Summary for Week 12 Summer 2008 =========================================================================== ------------------------ Lower bounds on problems [ chapter 8.1 ] ------------------------ A _problem_ in computer science is an input/output relationship: given some input, determine a/the correct output or answer. - example: the sorting problem Given Input: a sequence of n numbers (a_1, ..., a_n) Find Output: a permutation (reordering) (a_1', ..., a_n') of this sequence such that a_1' <= a_2' <= ... <= a_n' Review of running times for algorithms - an algorithm solves some problem (at least a correct algorithm does) - for each input X, the algorithm will take some time T(X) to compute its answer - often we wish to know the worst-case running time of the algorithm over all inputs of a certain size n T(n) = max { T(X) | X is an input of size n } - a smaller worst-case running time means a better performance guarantee (you're guaranteed an answer faster) - when do we know we have the BEST algorithm? i.e., can you prove that your algorithm has the lowest possible worst-case running time? To answer this question, we can't just look at your algorithm... we need to look at ALL algorithms that solve the problem (even those that we haven't invented yet!) - let A be the set of all algorithms that solve a particular problem P - let us define a function for the best worst-case running time (on inputs of size n) over all algorithms in A: BWC(n) = min { T_a(n) | a in A } - we can lower bound this function (asymptotically) - i.e., a function LB(n) such that LB(n) <= T_a(n) for all a in A - this is a lower bound for the problem (a lower bound on the worst-case time complexity of any algorithm that solves this problem) Note that it might be hard to argue about ALL possible problems. Often we must restrict ourselves to looking at a particular type of algorithm. Let's look at an example. Lower bound on (comparison-based) sorting ----------------------------------------- A _comparison-based_ algorithm is an algorithm where the behaviour of the algorithm is based only on the _comparisons_ between elements. We will examine _lower_bounds_ for comparison-based _sorting_ (lower bounds for searching will be covered in tutorial). In a comparison-based sort, we only use comparisons between elements to gain information about the order of a sequence. Given two elements a_i and a_j, we perform one of the tests: a_i < a_j, a_i <= a_j, a_i = a_j, a_i => a_j, or a_i > a_j. We can say that each test returns one of the following outcomes: (<=,>), (<,=>), (<=,=>), (<, =, >), (=, not =) We can express a comparison sort as a _decision_tree_ Example: Let's look at a _particular_ decision tree for sorting 3 elements. -internal nodes = comparisons -leaf nodes = final sorted orders -each branch is labelled with the outcome of a comparison Sort A, B, C: A:B <= > B:C etc. <= > A:C <= > Note: This is only a particular decision tree for this sort.. there are other possibilities. The decision tree depends on the algorithm that we are using. Observe: -this decision tree has 6 leaves -every permutation of the elements is in a leaf node Important Fact: The length of the longest path from the root of the decision tree to a leaf represents the worst-case number of comparisons that the sorting algorithm performs. => worst-case number of comparisons = the height of the decision tree How can we find the algorithm with the SMALLEST worst-case number of comparisons? We find the decision tree with the SMALLEST height. In other words: to find the worst-case running time of the "best" algorithm (i.e., the one with the smallest worst-case number of comparisons), we want to find a LOWER BOUND on the height of the decision trees. Fact 1: the number of ways to sort n numbers is n! (each permutation of the numbers is possible) This implies: the number of leaves in the decision tree is at least n! (there may be duplicate leaves) Fact 2: There are at most 3 outcomes (branches) at any level of the tree: (<, =, >) [Recall the other possible outcomes are: (<=,>), (<,=>), (<=,=>), (=, not =)] Fact 3: A ternary tree of height h has at most 3^h leaves (In other words: a tree with at least 3^h leaves must have height at least h) We can conclude: the decision tree for sorting has height at least log_3 n! Since log_3 n! in Theta(n log n), the height of the tree is in Omega(n log n). => the worst-case number of comparisons for ANY comparison-based sorting algorithm is in Omega(log n). Aside: How do we show that log_3 n! in Theta(log n) First, recall that log_2 n! is in Theta(n log_2 n). We proved this earlier in the term. [You should be able to show that log_3 n! is in Theta(n log_3 n).] Now let k = log_3 n => 3^k = n (take both sides as the exponent of 3) => k log_2 3 = log_2 n (take the log_2 of both sides) => log_3 n = log_2 n / log_2 3 (replace k with log_3 n, and divide both sides by log_2 3) Therefore log_3 n is in Theta(log_2 n). Therefore log_3 n! in Theta(log n). --------------------------------- Lower bounds: adversary arguments --------------------------------- Let's find a lower bound on the worst-case time needed to compute the following problem: Is there an element that appears exactly 5 times and another element that appears exactly 7 times in an n-element array of integers? Do you know an algorithm to solve this problem? Step 1: Sort all the elements Step 2: Scan through the elements and count whether one element appears 5 times and another appears 7 times. The total worst-case time of this algorithm is in O(n log n) So, either the lower bound on the worst-case time needed is Omega(n log n) or it is something smaller. Let's compute the lower bound using the decision tree method. Fact 1: there are two possible solutions to this problem: yes and no This implies: the number of leaves in the decision tree is at least 2 (there may be duplicate leaves) Fact 2: There are at most 3 outcomes (branches) at any level of the tree: (<, =, >) We can conclude: the decision tree for sorting has height at least log_3 2 Since log_3 2 in Omega(1), we have a lower bound of Omega(1). BUT, our lower and upper bounds don't match, and there doesn't seem to be a constant time algorithm that solves this problem. What do we do? Try a different method: Adversary method Algorithm: -asks a question -tries to reduce choices as quickly as possible (i.e., tries to learn the input) Adversary: -answers questions such that at least one input is consistent with the answers -tries to keep the legal set of inputs as big as possible (i.e., it ties to make you do as much work as possible) Example: The game "Guess Who?" I brought it in, and we talked about strategies for winning the game if you didn't need to pick a particular person (i.e., input) at the beginning of the game. We will use an adversary argument to solve the problem above. Suppose the algorithm only take n-1 steps. Then, it only reads n-1 spots in the array The adversary let the algorithm read the following input: x x x x x y y y y y y y . . . . The algorithm is correct, and saw 5 x's and 7 y's so it answers 'yes'. BUT it did not see one of the input values Case 1: The algorithm did not see a '.' value If the adversary changes the input so the '.' becomes an 'x', we have: x x x x x y y y y y y y . . x . The algorithm behaves the same way, and answers incorrectly! Case 2: The algorithm did not see a 'x' Case 3: The algorithm did not see a 'y' For case each, we can change the input so that the algorithm answers incorrectly! Therefore, we conclude that the algorithm needs to take at least n steps => a lower bound for the worst-case time for ANY algorithm to solve this problem is in Omega(n) This is better than the Omega(1) lower bound from the decision tree method. Something to think about: can we find an algorithm that takes linear time, or is our lower bound still too low?