1 growth of functions cs/apma 202 rosen section 2.2 aaron bloomfield

11

Growth of FunctionsGrowth of Functions

CS/APMA 202CS/APMA 202

Rosen section 2.2Rosen section 2.2

Aaron BloomfieldAaron Bloomfield

22

How does one measure algorithmsHow does one measure algorithms

We can time how long it takes a computerWe can time how long it takes a computer What if the computer is doing other things?What if the computer is doing other things? And what happens if you get a faster And what happens if you get a faster

computer?computer?A 3 Ghz Windows machine chip will run an A 3 Ghz Windows machine chip will run an algorithm at a different speed than a 3 Ghz algorithm at a different speed than a 3 Ghz MacintoshMacintosh

So that idea didn’t work out well…So that idea didn’t work out well…

33


We can measure how many machine We can measure how many machine instructions an algorithm takesinstructions an algorithm takes Different CPUs will require different amount of Different CPUs will require different amount of

machine instructions for the same algorithmmachine instructions for the same algorithm

So that idea didn’t work out well…So that idea didn’t work out well…

44


We can loosely define a “step” as a single We can loosely define a “step” as a single computer operationcomputer operation A comparison, an assignment, etc.A comparison, an assignment, etc. Regardless of how many machine instructions it Regardless of how many machine instructions it

translates intotranslates into

This allows us to put algorithms into broad This allows us to put algorithms into broad categories of efficientnesscategories of efficientness An efficient algorithm on a slow computer will An efficient algorithm on a slow computer will alwaysalways

beat an inefficient algorithm on a fast computerbeat an inefficient algorithm on a fast computer

55

Bubble sort running timeBubble sort running time

The bubble step take (The bubble step take (nn22--nn)/2 “steps”)/2 “steps”

Let’s say the bubble sort takes the following Let’s say the bubble sort takes the following number of steps on specific CPUs:number of steps on specific CPUs: Intel Pentium IV CPU:Intel Pentium IV CPU: 58*( 58*(nn22--nn)/2)/2 Motorola CPU: 84.4*(Motorola CPU: 84.4*(nn22-2-2nn)/2)/2 Intel Pentium V CPU: 44*(Intel Pentium V CPU: 44*(nn22--nn)/2)/2

Notice that each has an Notice that each has an nn22 term term As As nn increases, the other terms will drop out increases, the other terms will drop out

66

Bubble sort running timeBubble sort running time

This leaves us with:This leaves us with: Intel Pentium IV CPU:Intel Pentium IV CPU: 29 29nn22

Motorola CPU: 42.2Motorola CPU: 42.2nn22

Intel Pentium V CPU: 22Intel Pentium V CPU: 22nn22

As processors change, the constants will As processors change, the constants will always changealways change The exponent on The exponent on nn will not will not

Thus, we can’t care about the constantsThus, we can’t care about the constants

77

An aside: inequalitiesAn aside: inequalities

If you have a inequality you need to show:If you have a inequality you need to show:xx < < yy

You can replace the lesser side with something You can replace the lesser side with something greater:greater:

xx+1 < +1 < yy

If you can still show this to be true, then the If you can still show this to be true, then the original inequality is trueoriginal inequality is true

Consider showing that 15 < 20Consider showing that 15 < 20 You can replace 15 with 16, and then show that 16 < You can replace 15 with 16, and then show that 16 <

20. Because 15 < 16, and 16 < 20, then 15 < 2020. Because 15 < 16, and 16 < 20, then 15 < 20

88


If you have a inequality you need to show:If you have a inequality you need to show:xx < < yy

You can replace the greater side with something You can replace the greater side with something lesser:lesser:

xx < < yy-1-1

If you can still show this to be true, then the If you can still show this to be true, then the original inequality is trueoriginal inequality is true

Consider showing that 15 < 20Consider showing that 15 < 20 You can replace 20 with 19, and then show that 15 < You can replace 20 with 19, and then show that 15 <

19. Because 15 < 19, and 19 < 20, then 15 < 2019. Because 15 < 19, and 19 < 20, then 15 < 20

99


What if you do such a replacement and What if you do such a replacement and can’t show anything?can’t show anything? Then you can’t say anything about the original Then you can’t say anything about the original

inequalityinequality

Consider showing that 15 < 20Consider showing that 15 < 20 You can replace 20 with 10You can replace 20 with 10 But you can’t show that 15 < 10But you can’t show that 15 < 10 So you can’t say anything one way or the So you can’t say anything one way or the

other about the original inequalityother about the original inequality

10

Quick surveyQuick survey

I felt I understand running times and I felt I understand running times and inequality manipulation…inequality manipulation…

a)a) Very wellVery well

b)b) With some review, I’ll be goodWith some review, I’ll be good

c)c) Not reallyNot really

d)d) Not at allNot at all

1111

BiologyBiology

PhysicsPhysics

InterdisciplinaryInterdisciplinary ChemistryChemistry MathematicsMathematics LiteratureLiterature

PeacePeace

HygieneHygiene EconomicsEconomics

MedicineMedicine

The 2002 Ig Nobel PrizesThe 2002 Ig Nobel Prizes““Courtship behavior of ostriches towards humans underCourtship behavior of ostriches towards humans under

farming conditions in Britain”farming conditions in Britain”

““Demonstration of the exponential decay law using beer Demonstration of the exponential decay law using beer

froth”froth”

A comprehensive study of human belly button lintA comprehensive study of human belly button lint

Creating a four-legged periodic tableCreating a four-legged periodic table

““Estimation of the surface area of African elephants”Estimation of the surface area of African elephants”

““The effects of pre-existing inappropriate highlighting on The effects of pre-existing inappropriate highlighting on

reading comprehension”reading comprehension”

For creating Bow-lingual, a computerized dog-to-human For creating Bow-lingual, a computerized dog-to-human

translation devicetranslation device

For creating a washing machine for cats and dogsFor creating a washing machine for cats and dogs

Enron et. al. for applying imaginary numbers to the Enron et. al. for applying imaginary numbers to the

business worldbusiness world

““Scrotal asymmetry in man in ancient sculpture”Scrotal asymmetry in man in ancient sculpture”

1313

Review of last timeReview of last time

SearchesSearches Linear: Linear: nn steps steps Binary: logBinary: log22 nn steps steps Binary search is about as fast as you can getBinary search is about as fast as you can get

SortsSorts Bubble: Bubble: nn22 steps steps Insertion: Insertion: nn22 steps steps There are other, more efficient, sorting techniquesThere are other, more efficient, sorting techniques

In principle, the fastest are heap sort, quick sort, and merge In principle, the fastest are heap sort, quick sort, and merge sortsortThese each take take These each take take nn * log * log22 nn steps stepsIn practice, quick sort is the fastest, followed by merge sortIn practice, quick sort is the fastest, followed by merge sort

1414

Big-Oh notationBig-Oh notation

Let Let bb((xx) be the bubble sort algorithm) be the bubble sort algorithmWe say We say bb((xx) is ) is OO((nn22)) This is read as “This is read as “bb((xx) is big-oh ) is big-oh nn22”” This means that the input size increases, the running This means that the input size increases, the running

time of the bubble sort will increase proportional to the time of the bubble sort will increase proportional to the square of the input sizesquare of the input size

In other words, by some constant times In other words, by some constant times nn22

Let Let ll((xx) be the linear (or sequential) search ) be the linear (or sequential) search algorithmalgorithmWe say We say ll((xx) is ) is OO((nn)) Meaning the running time of the linear search Meaning the running time of the linear search

increases directly proportional to the input sizeincreases directly proportional to the input size

1515

Big-Oh notationBig-Oh notation

Consider: Consider: bb((xx) is ) is OO((nn22)) That means that That means that bb((xx)’s running time is less )’s running time is less

than (or equal to) some constant times than (or equal to) some constant times nn22

Consider: Consider: ll((xx) is ) is OO((nn)) That means that That means that ll((xx)’s running time is less )’s running time is less

than (or equal to) some constant times than (or equal to) some constant times nn

1616

Big-Oh proofsBig-Oh proofs

Show that Show that ff((xx) = ) = xx22 + 2 + 2xx + 1 is + 1 is OO((xx22)) In other words, show that In other words, show that xx22 + 2 + 2xx + 1 + 1 ≤≤ cc**xx22

Where Where cc is some constant is some constantFor input size greater than some For input size greater than some xx

We know that 2We know that 2xx22 ≥≥ 2 2xx whenever x whenever x ≥≥ 1 1And we know that And we know that xx22 ≥≥ 11 whenever x whenever x ≥≥ 1 1So we replace 2So we replace 2xx+1 with 3+1 with 3xx22

We then end up with We then end up with xx22 + 3 + 3xx22 = 4 = 4xx22

This yields 4This yields 4xx22 ≤≤ cc**xx22

This, for input sizes 1 or greater, when the constant is 4 This, for input sizes 1 or greater, when the constant is 4 or greater, or greater, ff((xx) is O() is O(xx22))We could have chosen values for We could have chosen values for cc and and xx that were that were differentdifferent

1717

Big-Oh proofsBig-Oh proofs

1818

Rosen, section 2.2, question 2(b)Rosen, section 2.2, question 2(b)

Show that Show that ff((xx) = ) = xx22 + 1000 is + 1000 is OO((xx22)) In other words, show that In other words, show that xx22 + 1000 + 1000 ≤≤ c* c*xx22

We know that We know that xx22 > 1000 whenever > 1000 whenever xx > 31 > 31 Thus, we replace 1000 with Thus, we replace 1000 with xx22

This yields 2This yields 2xx22 ≤≤ c* c*xx22

Thus, Thus, ff((xx) is ) is OO((xx22) for all x > 31 when c ) for all x > 31 when c ≥≥ 2 2

1919

Rosen, section 2.2, question 1(a)Rosen, section 2.2, question 1(a)

Show that Show that ff((xx) = 3) = 3xx+7 is +7 is OO((xx)) In other words, show that 3In other words, show that 3xx+7 +7 ≤≤ c* c*xx

We know that We know that xx > 7 whenever > 7 whenever xx > 7 > 7 Duh!Duh! So we replace 7 with So we replace 7 with xx This yields 4This yields 4xx ≤≤ cc**xx

Thus, Thus, ff((xx) is ) is OO((xx) for all x > 7 when c ) for all x > 7 when c ≥≥ 4 4

20


I felt I understand (more or less) Big-I felt I understand (more or less) Big-Oh proofs…Oh proofs…





2121

Today’s demotivatorsToday’s demotivators

2222

A variant of the last questionA variant of the last question

Show that Show that ff((xx) = 3) = 3xx+7 is +7 is OO((xx22)) In other words, show that 3In other words, show that 3xx+7 +7 ≤≤ c* c*xx22

We know that We know that xx > 7 whenever > 7 whenever xx > 7 > 7 Duh!Duh! So we replace 7 with So we replace 7 with xx This yields 4This yields 4xx < < cc**xx22

This will also be true for x > 7 when This will also be true for x > 7 when cc ≥≥ 1 1

Thus, Thus, ff((xx) is ) is OO((xx22) for all x > 7 when c ) for all x > 7 when c ≥≥ 1 1

2323

What that meansWhat that means

If a function is If a function is OO((xx)) Then it is also Then it is also OO((xx22)) And it is also And it is also OO((xx33))

Meaning a Meaning a OO((xx) function will grow at a ) function will grow at a slowerslower or equal to the rate or equal to the rate xx, , xx22, , xx33, etc., etc.

2424

Function growth ratesFunction growth rates

For input size n = 1000For input size n = 1000

O(1)O(1) 11O(log n)O(log n) ≈10≈10O(n)O(n) 101033

O(n log n)O(n log n) ≈10≈1044

O(nO(n22)) 101066

O(nO(n33)) 101099

O(nO(n44)) 10101212

O(nO(ncc)) 10103*c3*c cc is a consant is a consant22nn ≈10≈10301301

n!n! ≈10≈1025682568

nnnn 101030003000

Many interesting problems fall into this category

2525

Function growth ratesFunction growth rates

Logarithmic scale!

2626

Integer factorizationInteger factorization

Factoring a composite number into it’s Factoring a composite number into it’s component primes is component primes is OO(2(2nn))

This, if we choose 2048 bit numbers (as in This, if we choose 2048 bit numbers (as in RSA keys), it takes 2RSA keys), it takes 220482048 steps steps That’s about 10That’s about 10617617 steps! steps!

2727

Formal Big-Oh definitionFormal Big-Oh definition

Let Let ff and and gg be functions. We say that be functions. We say that ff((xx) ) is is OO((gg((xx)) if there are constants )) if there are constants cc and and kk such thatsuch that

|f(x)| ≤ |f(x)| ≤ C C ||gg((xx)|)|

whenever whenever xx > > kk

2828

Formal Big-Oh definitionFormal Big-Oh definition

2929

A note on Big-Oh notationA note on Big-Oh notation

Assume that a function Assume that a function ff((xx) is ) is OO((gg((xx))))

It is sometimes written as It is sometimes written as ff((xx) ) == OO((gg((xx)))) However, this is not a proper equality!However, this is not a proper equality! It’s really saying that |f(x)| ≤ It’s really saying that |f(x)| ≤ C C ||gg((xx)|)|

In this class, we will write it as In this class, we will write it as ff((xx) ) is is OO((gg((xx))))

3030

The growth of combinations of The growth of combinations of functionsfunctions

Ignore this partIgnore this part

3131

Big-omega and Big-thetaBig-omega and Big-theta

Ignore this partIgnore this part

3232

NP CompletenessNP Completeness

Not in the textbook – this is additional materialNot in the textbook – this is additional material

A full discussion of NP completeness takes 3 A full discussion of NP completeness takes 3 hours for somebody who already has a CS hours for somebody who already has a CS degreedegree We are going to do the 15 minute version of itWe are going to do the 15 minute version of it

Any term of the form Any term of the form nncc, where , where cc is a constant, is is a constant, is a polynomiala polynomial Thus, any function that is O(Thus, any function that is O(nncc) is a polynomial-time ) is a polynomial-time

functionfunction 22nn, , nn!, !, nnnn are not polynomial functions are not polynomial functions

3333

SatisfiabilitySatisfiability

Consider a Boolean expression of the Consider a Boolean expression of the form:form:((xx11 xx22 xx33) ) ( (xx22 xx33 xx44) ) ( (xx11 xx44 xx55)) This is a conjunction of disjunctionsThis is a conjunction of disjunctions

Is such an equation satisfiable?Is such an equation satisfiable? In other words, can you assign truth values to In other words, can you assign truth values to

all the all the xxii’s such that the equation is true?’s such that the equation is true?

3434

SatisfiabilitySatisfiability

If given a solution, it is easy to check if such a solution If given a solution, it is easy to check if such a solution worksworks

Plug in the values – this can be done quickly, even by handPlug in the values – this can be done quickly, even by hand

However, there is no known efficient way to find such a However, there is no known efficient way to find such a solutionsolution

The only definitive way to do so is to try all possible values for The only definitive way to do so is to try all possible values for the the nn Boolean variables Boolean variables

That means this is That means this is OO(2(2nn)!)! Thus it is not a polynomial time functionThus it is not a polynomial time function

NP stands for “Not Polynomial”NP stands for “Not Polynomial”

Cook’s theorem (1971) states that SAT is NP-completeCook’s theorem (1971) states that SAT is NP-complete There still may be an efficient way to solve it, though!There still may be an efficient way to solve it, though!

3535


There are hundreds of NP complete problemsThere are hundreds of NP complete problems It has been shown that if you can solve one of them It has been shown that if you can solve one of them

efficiently, then you can solve them allefficiently, then you can solve them all Example: the traveling salesman problemExample: the traveling salesman problem

Given a number of cities and the costs of traveling from any Given a number of cities and the costs of traveling from any city to any other city, what is the cheapest round-trip route city to any other city, what is the cheapest round-trip route that visits each city once and then returns to the starting city? that visits each city once and then returns to the starting city?

Not all algorithms that are Not all algorithms that are OO(2(2nn) are NP ) are NP completecomplete In particular, integer factorization (also In particular, integer factorization (also OO(2(2nn)) is not )) is not

thought to be NP completethought to be NP complete

3636


It is “widely believed” that there is no efficient solution to It is “widely believed” that there is no efficient solution to NP complete problemsNP complete problems

In other words, everybody has that beliefIn other words, everybody has that belief

If you could solve an NP complete problem in polynomial If you could solve an NP complete problem in polynomial time, you would be showing that P = NPtime, you would be showing that P = NP

If this were possible, it would be like proving that If this were possible, it would be like proving that Newton’s or Einstein’s laws of physics were wrongNewton’s or Einstein’s laws of physics were wrong

In summary: In summary: NP complete problems are very difficult to solve, but easy to NP complete problems are very difficult to solve, but easy to

check the solutions ofcheck the solutions of It is believed that there is no efficient way to solve themIt is believed that there is no efficient way to solve them

37


I sorta kinda get the hang of NP I sorta kinda get the hang of NP completenesscompleteness





3838

Star Wars: Episode III trailerStar Wars: Episode III trailer

No, really!No, really!

39


I felt I understood the material in this I felt I understood the material in this slide set…slide set…





40


The pace of the lecture for this The pace of the lecture for this slide set was…slide set was…

a)a) FastFast

b)b) About rightAbout right

c)c) A little slowA little slow

d)d) Too slowToo slow

41


How interesting was the material in How interesting was the material in this slide set? Be honest!this slide set? Be honest!

a)a) Wow! That was SOOOOOO cool!Wow! That was SOOOOOO cool!

b)b) Somewhat interestingSomewhat interesting

c)c) Rather bortingRather borting

d)d) ZzzzzzzzzzzZzzzzzzzzzz

1 growth of functions cs/apma 202 rosen section 2.2 aaron bloomfield

Documents