comp3121 2 basic tools for analysis of algorithms

8/12/2019 COMP3121 2 Basic Tools for Analysis of Algorithms

1/23

Topic 1:

Basic tools for analysis of algorithms

Assignment: please read pages: 41-90(23-76 in the first edition)

Recurrences

Recurrences are recursive definitions, i.e.

definitions that specify values of a functionin terms of values of the same function for

smaller arguments, like:

(n+1)! = (n+1)n!

f(n+1) = f(n) + f(n-1)f(n) = f( n/2) + n

Recurrences naturally arise when estimating

the run time of recursive algorithms. Forexample, recall that theMergeSorthas the

following pseudo-code:

1


2/23

Merge-Sort(A,p,r)

1if p < r

2 then q

(p+r)/23 Merge-Sort(A,p,q)4 Merge-Sort(A,q+1,r)

5 Merge(A,p,q,r)

TheMergeprocedure is running in linear

time, i.e., in time k nfor some k.

Thus, as we have shown, the run time of

MergeSortsatisfies recurrence formula

T(n)=2T(n/2)+ C n

To make things a bit simpler we will assumen is a power of two; in the CLRS textbook it

is shown that thats OK. Thus we get the

recurrence formula:

T(n)=2T(n/2)+ C n

There are ways to solve some recurrences

explicitly, but they all tend to be somewhat

tricky.

2


3/23

However, in estimating the efficiency of an

algorithm wedo notneed the solution of arecurrence formula, but we only need to find

the growth rateof the solutioni.e., its

asymptotic behavior.

The master method providesan estimate of

the growth rateof the solution forrecurrences of the form

T(n) = a T(n/b) + f(n)

for certain classes of a, band overhead

functions f(n), without having to solve

these recurrences. This is what the Master

Theorem is all about.

3


4/23

If T(n) is interpreted as the number of steps

needed to execute an algorithm for an input

of length n, this recurrence corresponds to adivide and conquermethod, in which a

problem of size nis divided into aproblems

of size n/b, where a, bare positive constants:

T(n) = a T(n/b) + f(n)

number of sub-problem price for dividing the

sub-problems size problem and combining

the solutions

(overhead)

Clearly, in order to have a recursion at all, we

must have n/b< n; i.e., b > 1.

Also, since acorresponds to the number of

problems of smaller size, we are interested

only in cases with a r1.

Function f(n) is the overhead, i.e. the cost of

dividing the problem of size ninto asub-

problems of size n/band then putting the

results of sub-problems together.

4


5/23

The Master Theorem:

Let a, bbe constants such that a r1 and

b >1, letf(n)be a function and let T(n)be

defined on natural numbers by the

recurrence T(n)=a T(n/b) + f(n)Then T(n) can be bounded asymptotically asfollows :

1) If f(n)=O(nlogba-e) for somee > 0,

then: T(n)=Q(nlogba)

Thus, iff(n)does not grow faster than the

polynomial nlogb a-e of degree strictly

smallerthan logba then the sollution T(n)

grows as the polynomial nlogb a.

5


6/23

2) If f(n)=Q(nlogba)

then : T(n)=Q(nlogbalogn)

Thus, iff(n) grows exactly as fast asthe

polynomial nlogb a then the solution T(n)

grows exactly as fast as nlogb alog n.

6


7/23

3) If f(n) = W(nlogba+e) for somee > 0,

and if af(n/b)bc f(n)for some constant c


8/23

Informally speaking,

i) iff(n)growssubstantially slowerthan

the polynomial nlogb a then the solution

T(n) grows exactly as nlogb a ;

ii) iff(n) growsas fast as the polynomial

nlogb a then the sollution T(n) growsexactly as n

logb alogn;

iii) finally, if f(n)growssubstantially

faster than the polynomial nlogb a and the

extraconditiona f(n/b)bc f(n)is satisfied

for some c


9/23

Dumb Examples (better ones later):

1. Let T(n) = 9 T(n/3) + n. Then a = 9 andb = 3, so nlog

39=n

2. Since f(n) = n =

O(n2-.5

) = O(n1.5

)the first case applies

with e= .5 and we conclude that T(n) =

n2.

2. If T(n) = T(2/3n) + 1 then a = 1 and b= 3/ 2, so n

log3/2

1=n

0=1. Thus, both

f(n)=1 and nlog

ba=1 which implies

f(n)=Q(nlogb a) and so the second case

applies; consequently:

T(n)=Q(nlogb a logn)=Q(n

0logn)=Q(logn)

3. Consider now T(n) = 3T(n/4)+ n.Then

a=3and b=4and so nlog

ba= n

log43n

0.79.

Thus,f(n)= n = W(n.79+.1

)W(n0.89

). Also,

a f(n/b )=3 f(n/4 ) = 3 n/4=3/4 n=3/4 f(n)=.75n < .8 n = .8 f(n)

Consequently by case 3, T(n)=Q(n).

9


10/23

4. Finally, consider recurrence

T(n)=2T(n/2)+n log n

In this case a=2, b=2,f(n) = nlogn. Thus,

nlogb a=n

log2 2= n. While f(n)= n log n >n,

but for no e f(n)=n log n >n1+e

for

sufficiently large n, so the third case doesnot apply and thus wecannot use the

master theorem.

Homework: Use basic calculus to show that

for every e there isN such that for all n > N

ne>log n.

10


11/23

Real examples:

Analysis of Merge Sort

As we saw, Merge Sort running time is

given by the recurrence relation

T(n)=2T(n/2)+ C n

Thus, for the Master Theorem we have:

a=2, b=2, f(n)=Cn

Since

nlogba = n

log22 = n1= n

we get that the second case holds because

f(n)=Cn =(n)=(n

logba)

Thus, the solution of the recurrence satisfies

T(n)=(nlogbalog n)=(n

log n)

11


12/23

Real examples:

Analysis of the Karatsuba algorithm for

integer multiplication

The Karatsuba algorithmrunning time is

given by the recurrence relation

T(n)=3T(n/2)+ C n

Thus, for the Master Theorem we have:

a=3, b=2, f(n)=Cn

Since

nlogba = n

log23 = n1.58

we get that the first case holds, because

f(n)=Cn =O(n1.48

)=O(n

1.58 - .1)

=

O (n

log23-)

Thus, the solution of the recurrence satisfies

T(n)=(nlogba)= (n

log23)=(n

1.58)

12


13/23

Determining the running time of an

algorithm seems to be hard because it

requires knowing difficult theorems. Whybother? Why not just write a correct

program and not worry how fast it is?

n 16 256 65536

nlogn 64 2048 1048576n

2256 65536 4294967296

n3

4096 16777216 281474976710656

2n

65536 1077

1019728

The above table shows that how fastan

algorithm is, often determines in practice if

an algorithm will run at all or notbecause it

would take too much time to finish!!!

Why bother proving the master theoremand not just take it on faith?

13


14/23

For a practical reason :

We areinterested intechniquesused in thisproof : we want to acquire skills inhow to

countthe total number of steps of a

recursive procedure,how to do sumationsof

various sums which will often pop up in our

analyses of algorithms etc. Themethodsof

the proof will keep appearing on manyoccasions.

Thus, by mastering the proof we will get

used to concepts involved and learn the

machinery used in estimating the run time of

manyalgorithms.

The same machinery can be used to estimate

the run time of recursive algorithms which

you might be designing yourselves!

14


15/23

Proof of the Master Theorem

We will prove the Master Theorem only for numbers which

are powers of b. The proof for an arbitrary number followsfrom the properties of the floor and ceiling functions; recall

2.99 = 2 and 2.01 = 3. It does not add anythinginteresting from the prospective of algorithms.

Proof : (concentrate on the method, NOT on calculations !!)

Step 1.(unwinding recursion):

Substitute m in T(m) = aT(m/b) + f(m)by

n/bjfor all 1ji where i = logb n (for

simplicity we assume nis a power of b.

Thus, assume that T(m) = aT(m/b) + f(m)

for all m.Then by substituting m in this

equality by n,n/b, n/b2and so on up ton/b

i-1,

we get : T(n) = aT(n/b) + f(n)

T(n/b) = aT(n/b2) + f(n/b)

T(n/b2

) = aT(n/b3

) + f(n/b2

).

.

.

T(n/bi-1) = aT(n/b

i) + f(n/b

i-1)

15


16/23

Step 2.Sucessive eliminationsof T(n/bk) in

order to replace the instances T(n/b

k

) of therecursively definedfunction T(n)by the

instancesf(n/bk) of the explicitely defined,

simpler functionf(n).

By consecutive replacements of T(n/bk) by

aT(n/bk+1) + f(n/bk) using equationsobtained at Step 1 we get :

T(n) = aT(n/b)+ f(n)=

a(aT(n/b2) + f(n/b)) + f(n)=

a2T(n/b

2)+a f(n/b) + f(n)=

a2(aT(n/b

3) + f(n/b

2)) + a f(n/b)+ f(n)=

a3T(n/b

3)+ a

2f(n/b

2)+a f(n/b)+ f(n)=

.

.

aiT(n/b

i)+a

i-1f(n/b

i-1)+a

i-2f(n/b

i-2)++ f(n)

Here we essentially unwind the recursion

untill we hit the base case T(n/bi)=T(1),

(because bi= n, i.e. i=logbn).

16


17/23

Thus,

T(n)=aiT(1) + a

i-1f(n/b

i-1)+a

i-2f(n/b

i-2)++ f(n)=

i summands

=

=

+=+1)(log

0

log1

0

)()1()()1(n

jb

njni

jb

njib

jb

j faTafaTa

Step 3:simplifying the coefficient in frontof T(1).

Since alogbn=n

logba we get:

=

+=1)(log

0

log)()1()(

n

jb

njab

j

b faTnnT

Thus, to estimate the behavior of T(n)we

have to estimate the behavior of the sum

=

=1log

0

)()(n

jb

njb

jfanS

Notice that in this sum )( jbnjfa is a product of members of an increasing

geometric progression aj

and values off(n/bj

) with inputsn/bj

that form adecreasing geometric progression. Thus, the behavior of the sum depends on

whether ajgrows faster thanf(n/b

j) decreases!! (we are interested only inf(n)

that is monotonically increasing, becausef(x) will stand for the overhead of

divide-and-conquer algorithms. ) This is the crux of Master Theorem!. We

now consider the growth rate off(n) accordingly, i.e., we replacef(n/bj)

with its asymptotic estimate.

17


18/23

Step 4:Replacing f(n/bj) by asymptotic

estimates. We proceed according to our

three cases.

Case 1: Assumef(m)=O(mlogba-e). Then

using properties of the big-oh, O(which are

easy to show) and by substitutingmin the

above byn/bi

we get f(n/bi)=O((n/b

i)

logb a-e),

and so

( )( ) ( )

( )

=

=

=

=

===

=

=

=

=

=

=

=

1log

0

log

1log

0

log1log

0

log

1log

0

log1log

0

log

1log

0

log1log

0

log

log)(

))(()()(

n

j

ja

n

j

j

aaba

n

j

j

b

aba

n

j

j

b

aan

j

a

b

nj

n

j

a

b

njn

jb

nj

b

b

b

b

b

ab

b

b

ab

b

b

bj

b

bj

b

j

bnO

nOnO

nOaO

OafanS

18


19/23

By using

nbq

qqqq nn

n b==++++

+

log

1

2 and11...1

we sum the geometric progression

and get

( )

=

1log

0

n

j

jb

b

=

=

1

1)(

loglog

b

bnOnS

na

b

b

( )aaa

a bbb

b nO

b

nnO

b

nnO

logloglog

log

11

1=

=

Consequently,

( )aan

j

b

nja bbb

jb nOTnfaTnnT

loglog1log

0

log)1()()1()( +=+=

=

19


20/23

However, adding to nlog

baT(1) something

that grows not faster than n

logba

results insomething that grows as fast as nlog

ba, i.e.

aaa bbb nnOTnnTlogloglog

)1()( =+=

This proves the first case.

20


21/23

Case 2: Assumef(n)=(nlog

ba). Then as in

the first case, we first estimate

( )

( )

( )nn

nn

na

afanS

b

a

n

j

an

j

j

aaa

n

j

j

b

aan

j

a

b

nj

n

j

a

b

njn

jb

nj

b

bb

bb

b

ab

b

b

bj

b

bj

b

j

log

1

)(

))(()()(

log

1log

0

log1log

0

log

1log

0

log1log

0

log

1log

0

log1log

0

log

=

=

=

=

===

=

=

=

=

=

=

Since logbn= logb2log2n we get

nnnSab

2

loglog)( =

Consequently, denoting log2nby lg n, we

get

( ) ( )nnnnTn

faTnnT

aaa

n

jbnja

bbb

b

jb

lglg)1(

)()1()(

logloglog

1log

0

log

=+

=+=

=

21


22/23

Case 3: In theassumption that

f(n/b)bc/a f(n) we substitute nby n/bk-1

to

getf(n/bk)bc/a f(n/bk-1) . Thus, we get:f(n/b

j)bc/a f(n/b

j-1)

f(n/bj-1

)bc/a f(n/bj-2

)

f(n/bj-2

)bc/a f(n/bj-3

)

...f(n)bc/a f(n/b)

By substitution, chaining these inequalities,

we get

f(n/bj)bc/a f(n/b

j-1)b(c/a)

2f(n/b

j-2) b

...b(c/a)jf(n),i.e. we get

f(n/bj)bc

j/a

jf(n)

22


23/23

Thus, we can estimate S(n)as follows

))((1

)()()()(

)()()(

0

1log

0

1log

0

1log

0

1log

0

nfOc

nfcnfcnfnfc

nfacafanS

j

jn

j

jn

j

j

n

jj

j

j

n

jbnj

bb

bb

j

=

==

==

=

=

=

=

=

Consequently, we get:

))(()1(

)()1()(

log

1log

0

log

nfOTn

faTnnT

a

n

jb

nja

b

b

j

b

+

=+=

=

By assumption in the third case

f(n) = W(nlogb a+e) for somee > 0, and so,

since f(n) grows much faster than nlog

ba,

))(())(()1()(log

nfOnfOTnnTab =+=

This finishes the proof for the third case and

thus we are done !

comp3121 2 basic tools for analysis of algorithms

Documents