===========================================================================
CSC 373H Lecture Summary for Week 7 Winter 2006
===========================================================================
------------------
Divide and Conquer
------------------
Integer multiplication.
- Problem: Multiply two integers x, y, given as sequences of bits
x_0,x_1,...,x_{n-1}; y_0,y_1,...,y_{n-1} (low-order bit first, i.e.,
x = x_{n-1} ... x_1 x_0 in binary, and similarly for y).
- Iterative algorithm: Multiply x by each bit of y, shifted
appropriately, then add the n results to each other.
Runtime = Theta(n^2) (n additions of up to 2n bits each).
- Idea: Let X_0 = x_{n/2-1} ... x_1 x_0 and X_1 = x_{n-1} ... x_{n/2},
in binary (n can always be even by adding additional 0's to the left of
x); define Y_0 and Y_1 similarly.
Then, x = 2^{n/2} X_1 + X_0 and y = 2^{n/2} Y_1 + Y_0, and we can write
x y = 2^n X_1 Y_1 + 2^{n/2} X_1 Y_0 + 2^{n/2} X_0 Y_1 + X_0 Y_0
How does this help? Original problem (compute x y) reduced to four
subproblems of half size (compute X_1 Y_1, X_1 Y_0, X_0 Y_1, X_0 Y_0),
together with some "shift" operations (multiplication by power of 2)
and binary additions.
This yields recursive algorithm directly.
Multiply(x, y): // x, y are arrays of size n
if n = 1:
return x * y // multiplication of 1-bit numbers
else:
set arrays X1, X0, Y1, Y0
p1 := Multiply(X1, Y1)
p2 := Multiply(X1, Y0)
p3 := Multiply(X0, Y1)
p4 := Multiply(X0, Y0)
return 2^n p1 + 2^{n/2} p2 + 2^{n/2} p3 + p4
Runtime? Recursive algorithm yields recurrence relation for worst-case
runtime T(n):
T(1) = Theta(1)
T(n) = 4 T(n/2) + Theta(n)
where 4 T(n/2) comes from the time spent executing the four recursive
calls, and Theta(n) comes from the time spent performing shifts and
binary additions.
Closed form?
Master Theorem.
- Let f be any nondecreasing function that satisfies the following
recurrence, with constants integer a > 0, rational b > 1, real d >= 0,
and integer k >= 1:
f(n) = Theta(1) if n <= k,
f(n) = a f(n/b) + Theta(n^d) if n > k.
For example, f(n) could represent the runtime of a recursive algorithm
that makes "a" recursive calls, each one to an input of size roughly
n/b (ignoring floors and ceilings), in addition to taking time
Theta(n^d) to perform work outside of the recursive calls.
Then, f(n) has the following closed-form asymptotic solution:
f(n) = Theta(n^{log_b a}) if a > b^d,
f(n) = Theta(n^d log n) if a = b^d,
f(n) = Theta(n^d) if a < b^d.
Integer multiplication (continued).
- Master Theorem applies to T(n), with a = 4, b = 2, d = 1. Since
a = 4 > 2 = b^d, we have T(n) = Theta(n^{log_2 4}) = Theta(n^2).
This is no better than simple iterative algorithm!
- Trick: using same notation as before, notice that
(X_1 + X_0) (Y_1 + Y_0) = X_1 Y_1 + X_1 Y_0 + X_0 Y_1 + X_0 Y_0.
This is almost correct expression, except for shifts, and it involves
only 1 multiplication instead of 4. Because terms X_1 Y_0 and X_0 Y_1
shift by same amount, we can use this to save one recursive call:
x y = 2^n X_1 Y_1 + X_0 Y_0
+ 2^{n/2} ( (X_1 + X_0) (Y_1 + Y_0) - X_1 Y_1 - X_0 Y_0 )
- This yields following recursive algorithm:
Multiply2(x, y):
if n = 1:
return x * y // multiplication of 1-bit numbers
else:
set arrays X1, X0, Y1, Y0
p1 := Multiply2(X1, Y1)
p2 := Multiply2(X1 + X_0, Y_1 + Y0)
p3 := Multiply2(X0, Y0)
return 2^n p1 + 2^{n/2} (p2 - p1 - p3) + p_3
with runtime T'(n) that satisfies:
T'(n) = Theta(1)
T'(n) = 3 T(n/2) + Theta(n)
The constant hidden by the term Theta(n) is larger than for the first
recursive algorithm (we perform more binary additions), but the Master
Theorem still applies with a = 3, b = 2, d = 1, which yields
T'(n) = Theta(n^{log_2 3}) = Theta(n^{1.58...}).
This is strictly better than previous Theta(n^2)!
- Picture of savings from 4 T(n/2) to 3 T(n/2).
- In practice, best known algorithm is "Fast Fourier Transform" (FFT)
algorithm -- a more complicated divide-and-conquer algorithm -- with
runtime Theta(n log n log log n).
Merge sort.
- Given array A:
1. split into two halves B,C;
2. sort B,C recursively;
3. merge B,C back into A (in linear time).
- Runtime:
T(1) = Theta(1)
T(n) = 2 T(n/2) + Theta(n)
Closed-form: T(n) = Theta(n log n) (a = 2 = 2^1 = b^d).
Quick sort.
- Given array A:
1. pick "pivot" element p;
2. partition A into B = [elements < p] and C = [elements > p],
in place (i.e., so that A = [B,p,C]);
3. sort B,C recursively, in place.
- Runtime depends on pivot picked at each step:
. worst-case degenerates to Theta(n^2);
. average-case is Theta(n log n).
Counting inversions.
- Given a_1,...,a_n, a permutation of [1,...,n], count the number of
inversions (pairs i < j with a_i > a_j).
- Application: comparative ranking.
- Solution 1: Consider each pair.
Runtime: Theta(n^2).
- Solution 2: (divide and conquer)
1. split input permutation A into halves B,C;
2. count inversions in B,C recursively;
3. count inversions between B,C, i.e., number of pairs of elements
b in B, c in C such that b > c;
4. return total number of inversions.
- Problem: step 3 takes time Theta(n^2).
- Solution 2': (make it possible to do step 3 in linear time)
1. split input permutation A into halves B,C;
2. SORT and count inversions in B,C recursively;
3. MERGE and count inversions between B,C, i.e.,
at each step of merge, comparing next element b in B, c in C:
. if b < c then b < all remaining elements in C,
so no more inversions involve b
. if c < b then c < all remaining elements in B,
so add remaining number of elements in B to inversion count
(note: number of inversions between B,C not affected by sorting
within each of B,C because all pairs b in B, c in C such that b > c
remain the same);
4. return total number of inversions.
- Extra sorting makes it possible to count inversions between B,C in
linear time (at the same time as merge step), and total time same as
Mergesort: Theta(n log n).