11

CSC401 – Analysis of Algorithms Lecture Notes 11

Divide-and-ConquerDivide-and-ConquerObjectives:Objectives: Introduce the Divide-and-conquer paradigmIntroduce the Divide-and-conquer paradigm Review the Merge-sort algorithmReview the Merge-sort algorithm Solve recurrence equationsSolve recurrence equations

– Iterative substitutionIterative substitution– Recursion treesRecursion trees– Guess-and-testGuess-and-test– The master methodThe master method

Discuss integer and matrix multiplicationsDiscuss integer and matrix multiplications

22

Divide-and-ConquerDivide-and-ConquerDivide-and conquerDivide-and conquer is a general algorithm design is a general algorithm design paradigm:paradigm:– DivideDivide: divide the input data : divide the input data SS in two or more disjoint in two or more disjoint

subsets subsets SS11, S, S22, …, …– RecurRecur: solve the subproblems recursively: solve the subproblems recursively– ConquerConquer: combine the solutions for : combine the solutions for SS11,, SS22, …, into a solution , …, into a solution

for for SS

The base case for the recursion are subproblems of The base case for the recursion are subproblems of constant sizeconstant sizeAnalysis can be done using Analysis can be done using recurrence equationsrecurrence equations

33

Merge-Sort ReviewMerge-Sort ReviewMerge-sort on an input Merge-sort on an input sequence sequence SS with with nn elements consists of elements consists of three steps:three steps:– DivideDivide: partition : partition SS into into

two sequences two sequences SS11 and and SS22 of about of about nn22 elements elements eacheach

– RecurRecur: recursively sort : recursively sort SS11 and and SS22

– ConquerConquer: merge : merge SS11 and and SS2 2 into a unique sorted into a unique sorted sequencesequence

Algorithm mergeSort(S, C)Input sequence S with n

elements, comparator C Output sequence S sorted

according to Cif S.size() > 1

(S1, S2) partition(S, n/2)

mergeSort(S1, C)

mergeSort(S2, C)

S merge(S1, S2)7 2 9 4 2 4 7 9

7 2 2 7 9 4 4 9

7 7 2 2 9 9 4 4

44

Recurrence Equation AnalysisRecurrence Equation AnalysisThe conquer step of merge-sort consists of merging The conquer step of merge-sort consists of merging two sorted sequences, each with two sorted sequences, each with nn22 elements and elements and implemented by means of a doubly linked list, takes implemented by means of a doubly linked list, takes at most at most bnbn steps, for some constant steps, for some constant bb..Likewise, the basis case (Likewise, the basis case (nn < 2) < 2) will take at will take at bb most most steps.steps.Therefore, if we let Therefore, if we let TT((nn) ) denote the running time of denote the running time of merge-sort:merge-sort:

We can therefore analyze the running time of merge-We can therefore analyze the running time of merge-sort by finding a sort by finding a closed form solutionclosed form solution to the above to the above equation.equation.– That is, a solution that has That is, a solution that has TT((nn) ) only on the left-hand side.only on the left-hand side.

2if)2/(2

2if )(

nbnnT

nbnT

55

Iterative SubstitutionIterative SubstitutionIn the iterative substitution, or “plug-and-chug,” In the iterative substitution, or “plug-and-chug,” technique, we iteratively apply the recurrence technique, we iteratively apply the recurrence equation to itself and see if we can find a patternequation to itself and see if we can find a pattern

Note that base, T(n)=b, case occurs when 2Note that base, T(n)=b, case occurs when 2ii=n. =n. That is, i = log n. That is, i = log n. So,So,Thus, T(n) is O(n log n).Thus, T(n) is O(n log n).

ibnnT

bnnT

bnnT

bnnT

bnnbnT

bnnTnT

ii

)2/(2

...

4)2/(2

3)2/(2

2)2/(2

))2/())2/(2(2

)2/(2)(

44

33

22

2

nbnbnnT log)(

66

The Recursion TreeThe Recursion TreeDraw the recursion tree for the recurrence relation Draw the recursion tree for the recurrence relation and look for a pattern:and look for a pattern:

depthdepth T’sT’s sizesize

00 11 nn

11 22 nn22

ii 22ii nn22ii

…… …… ……

2if)2/(2

2if )(

nbnnT

nbnT

timetime

bnbn

bnbn

bnbn

……

Total time = bn + bn log n(last level plus all previous levels)

77

Guess-and-Test MethodGuess-and-Test MethodIn the guess-and-test method, we guess a closed In the guess-and-test method, we guess a closed form solution and then try to prove it is true by form solution and then try to prove it is true by induction:induction:

Guess: T(n) < cnlogn.Guess: T(n) < cnlogn.

Wrong: we cannot make this last line be less than Wrong: we cannot make this last line be less than cn log ncn log n

nbncnncn

nbnncn

nbnnnc

nbnnTnT

loglog

log)2log(log

log))2/log()2/((2

log)2/(2)(

2iflog)2/(2

2if )(

nnbnnT

nbnT

88

Guess-and-Test Method, Part 2Guess-and-Test Method, Part 2Recall the recurrence equation:Recall the recurrence equation:

Guess #2: T(n) < cn logGuess #2: T(n) < cn log22 n. n.

– if c > b.if c > b.

So, T(n) is O(n logSo, T(n) is O(n log22 n). n).In general, to use this method, you need to have a In general, to use this method, you need to have a good guess and you need to be good at induction good guess and you need to be good at induction proofs.proofs.

ncn

nbncnncnncn

nbnncn

nbnnnc

nbnnTnT

2

2

2

2

log

loglog2log

log)2log(log

log))2/(log)2/((2

log)2/(2)(

2iflog)2/(2

2if )(

nnbnnT

nbnT

99

Master MethodMaster MethodMany divide-and-conquer recurrence Many divide-and-conquer recurrence equations have the form:equations have the form:

The Master Theorem:The Master Theorem:

dnnfbnaT

dncnT

if)()/(

if )(

.1 somefor )()/( provided

)),((is)(then),(is)(if 3.

)log(is)(then),log(is)(if 2.

)(is)(then),(is)(if 1.

log

1loglog

loglog

nfbnaf

nfnTnnf

nnnTnnnf

nnTnOnf

a

kaka

aa

b

bb

bb

1010

Master Method, ExamplesMaster Method, ExamplesThe form:The form:

The Master The Master Theorem:Theorem:

dnnfbnaT

dncnT

if)()/(

if )(

.1 somefor )()/( provided

)),((is)(then),(is)(if 3.

)log(is)(then),log(is)(if 2.

)(is)(then),(is)(if 1.

log

1loglog

loglog

nfbnaf

nfnTnnf

nnnTnnnf

nnTnOnf

a

kaka

aa

b

bb

bb

nnTnT )2/(4)(Example 1:Example 1:– Solution: a=4, b=2, logba=2, f(n)=n, so case 1

says T(n) is O(n2).

Example 2:– Solution: a=b=2, logba=1, f(n)=nlogn, so case 2

says T(n) is O(n log2 n).

nnnTnT log)2/(2)(

1111

Master Method, ExamplesMaster Method, Examples

The form:The form:

The Master The Master Theorem:Theorem:

dnnfbnaT

dncnT

if)()/(

if )(

.1 somefor )()/( provided

)),((is)(then),(is)(if 3.

)log(is)(then),log(is)(if 2.

)(is)(then),(is)(if 1.

log

1loglog

loglog

nfbnaf

nfnTnnf

nnnTnnnf

nnTnOnf

a

kaka

aa

b

bb

bb

Example 3:Example 3:– Solution: a=1, b=3, logba=0, f(n)=nlogn, so case 3 says

T(n) is O(nlogn).

Example 4:– Solution: a=2, b=8, logba=3, f(n)=n2, so case 1 says

T(n) is O(n3).

2)2/(8)( nnTnT

nnnTnT log)3/()(

1212

Master Method, ExamplesMaster Method, ExamplesThe form:The form:

The Master The Master Theorem:Theorem:

1)2/()( nTnT

dnnfbnaT

dncnT

if)()/(

if )(

.1 somefor )()/( provided

)),((is)(then),(is)(if 3.

)log(is)(then),log(is)(if 2.

)(is)(then),(is)(if 1.

log

1loglog

loglog

nfbnaf

nfnTnnf

nnnTnnnf

nnTnOnf

a

kaka

aa

b

bb

bb

Example 5:Example 5:– Solution: a=9, b=3, logba=2, f(n)=n3, so case 3 says

T(n) is O(n3).Example 6: (binary search)– Solution: a=1, b=2, logba=0, f(n)=1, so case 2 says

T(n) is O(log n).Example 7: (heap construction)– Solution: a=2, b=2, logba=1, f(n)=logn, so case 1

says T(n) is O(n).

3)3/(9)( nnTnT

nnTnT log)2/(2)(

1313

Iterative “Proof” of the Master TheoremIterative “Proof” of the Master TheoremUsing iterative substitution, let us see if we can Using iterative substitution, let us see if we can find a pattern:find a pattern:

We then distinguish the three cases asWe then distinguish the three cases as– The first term is dominantThe first term is dominant– Each part of the summation is equally dominantEach part of the summation is equally dominant– The summation is a geometric seriesThe summation is a geometric series

1)(log

0

log

1)(log

0

log

2233

22

2

)/()1(

)/()1(

. . .

)()/()/()/(

)()/()/(

))/())/((

)()/()(

n

i

iia

n

i

iin

b

b

b

b

bnfaTn

bnfaTa

nfbnafbnfabnTa

nfbnafbnTa

bnbnfbnaTa

nfbnaTnT

1414

Integer MultiplicationInteger MultiplicationAlgorithm: Multiply two n-bit integers I and Algorithm: Multiply two n-bit integers I and J.J.– Divide step: Split I and J into high-order and low-order Divide step: Split I and J into high-order and low-order

bitsbits

– We can then define I*J by multiplying the parts and We can then define I*J by multiplying the parts and adding:adding:

– So, T(n) = 4T(n/2) + n, which implies T(n) is O(nSo, T(n) = 4T(n/2) + n, which implies T(n) is O(n22).).– But that is no better than the algorithm we learned in But that is no better than the algorithm we learned in

grade school.grade school.

ln

h

ln

h

JJJ

III

2/

2/

2

2

lln

hln

lhn

hh

ln

hln

h

JIJIJIJI

JJIIJI

2/2/

2/2/

222

)2(*)2(*

1515

An Improved Integer Multiplication An Improved Integer Multiplication AlgorithmAlgorithm

Algorithm: Multiply two n-bit integers I and Algorithm: Multiply two n-bit integers I and J.J.– Divide step: Split I and J into high-order and low-order bitsDivide step: Split I and J into high-order and low-order bits

– Observe that there is a different way to multiply parts:Observe that there is a different way to multiply parts:

– So, T(n) = 3T(n/2) + n, which implies T(n) is O(nSo, T(n) = 3T(n/2) + n, which implies T(n) is O(nloglog22

33), by ), by the Master Theorem.the Master Theorem.

– Thus, T(n) is O(nThus, T(n) is O(n1.5851.585).).

ln

h

ln

h

JJJ

III

2/

2/

2

2

lln

hllhn

hh

lln

llhhhlhhlllhn

hh

lln

llhhhllhn

hh

JIJIJIJI

JIJIJIJIJIJIJIJI

JIJIJIJJIIJIJI

2/

2/

2/

2)(2

2])[(2

2]))([(2*

1616

Matrix Multiplication AlgorithmMatrix Multiplication AlgorithmProblem:Problem: Given n×n matrices X and Y, find their Given n×n matrices X and Y, find their product Z=X×Y, product Z=X×Y,

General algorithmGeneral algorithm: O(n: O(n33))

Strassen’s AlgorithmStrassen’s Algorithm::– Rewrite the matrix product as: Rewrite the matrix product as: – Where Where I=AE+BG, J=AF+BHI=AE+BG, J=AF+BH

K=CE+DG, L=CF+DHK=CE+DG, L=CF+DH– Define 7 submatrix products:Define 7 submatrix products:

S1=A(F-H), S2=(A+B)H, S3=(C+D)E, S4=D(G-E)S1=A(F-H), S2=(A+B)H, S3=(C+D)E, S4=D(G-E)

S5=(A+D)(E+H), S6=(B-D)(G+H), S7=(A-C)(E+F)S5=(A+D)(E+H), S6=(B-D)(G+H), S7=(A-C)(E+F)– Construct: Construct:

I=S4+S5+S6-S2, J=S1+S2, K=S3+S4, L=S1-S7-S3-I=S4+S5+S6-S2, J=S1+S2, K=S3+S4, L=S1-S7-S3-S5S5

– Running time: T(n) = 7T(n/2)+bnRunning time: T(n) = 7T(n/2)+bn22 for some constant b, for some constant b, which gives: O(nwhich gives: O(nlog7log7), better than O(n), better than O(n33))

1

0

],[],[],[n

k

jkYkiXjiZ

HG

FE

DC

BA

LK

JI