chapter 1

CHAPTER 1CHAPTER 1

Fundamental ToolsFundamental Tools

22

Definition of an algorithm Definition of an algorithm (from CPSC 171):(from CPSC 171):

An algorithm is a well-ordered collection of unambiguous and effectively computable operations that, when executed, produces a result and halts in a finite amount of time.

The textbook has a similar definition:

.

33

Textbook DefinitionsTextbook Definitions

• An algorithm is a step-by-step procedure for performing some task in a finite about of time.

• A data structure is a systematic way of organizing and accessing data.

44

Running Time (§1.1) Running Time (§1.1)

Most algorithms Most algorithms transform input objects transform input objects into output objects.into output objects.

The running time of an The running time of an algorithm typically grows algorithm typically grows with the input size.with the input size.

Average case time is Average case time is often difficult to often difficult to determine.determine.

We often focus on the We often focus on the worst case running time.worst case running time.• Easier to analyzeEasier to analyze• Crucial to applications such Crucial to applications such

as games, finance and as games, finance and roboticsrobotics

0

20

40

60

80

100

120

Runnin

g T

ime

1000 2000 3000 4000

Input Size

best caseaverage caseworst case

55

Experimental Studies (§ 1.6)Experimental Studies (§ 1.6)

Write a program Write a program implementing the implementing the algorithmalgorithm

Run the program with Run the program with inputs of varying size inputs of varying size and compositionand composition

Use a method like Use a method like System.currentTimeMillis()System.currentTimeMillis() to to get an accurate get an accurate measure of the actual measure of the actual running timerunning time

Plot the resultsPlot the results

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

0 50 100

Input Size

Tim

e (

ms)

66

Limitations of ExperimentsLimitations of Experiments

It is necessary to implement the algorithm, It is necessary to implement the algorithm, which may be difficultwhich may be difficult

Results may not be indicative of the running Results may not be indicative of the running time on other input not included in the time on other input not included in the experiment. experiment.

In order to compare two algorithms, the same In order to compare two algorithms, the same hardware and software environments must be hardware and software environments must be usedused

Different programmers have different abilities to Different programmers have different abilities to write “good, efficient” code.write “good, efficient” code.

We can’t examine all cases.We can’t examine all cases.

77

Theoretical AnalysisTheoretical Analysis Uses a high-level description of the Uses a high-level description of the

algorithm instead of an algorithm instead of an implementationimplementation

Characterizes running time as a Characterizes running time as a function of the input size, function of the input size, nn..

Takes into account all possible inputsTakes into account all possible inputs Allows us to evaluate the speed of an Allows us to evaluate the speed of an

algorithm independent of the algorithm independent of the hardware/software environmenthardware/software environment

88

Pseudocode (§1.1)Pseudocode (§1.1)

High-level description High-level description of an algorithmof an algorithm

More structured than More structured than English proseEnglish prose

Less detailed than a Less detailed than a programprogram

Preferred notation for Preferred notation for describing algorithmsdescribing algorithms

Hides program design Hides program design issuesissues

Algorithm arrayMax(A, n)Input array A of n integersOutput maximum element of A

currentMax A[0]for i 1 to n 1 do

if A[i] currentMax thencurrentMax A[i]

return currentMax

Example: find max element of an array

99

Pseudocode DetailsPseudocode Details

Control flowControl flow• ifif … … thenthen … [ … [elseelse …] …]

• whilewhile … … dodo … …

• repeatrepeat … … untiluntil … …

• forfor … … dodo … …

• Indentation replaces braces Indentation replaces braces Method declarationMethod declaration

AlgorithmAlgorithm methodmethod ( (argarg [, [, argarg…])…])

InputInput … …

OutputOutput … …

Method callMethod callvar.method var.method ((argarg [, [, argarg…])…])

Return valueReturn valuereturnreturn expressionexpression

ExpressionsExpressions AssignmentAssignment

(like (like in Java) in Java) Equality testingEquality testing

(like (like in Java) in Java)nn22 Superscripts and other Superscripts and other

mathematical formatting mathematical formatting allowedallowed

1010

The Random Access Machine The Random Access Machine (RAM) Model(RAM) Model

A A CPUCPU

An potentially An potentially unbounded bank of unbounded bank of memorymemory cells, each of cells, each of which can hold an which can hold an arbitrary number or arbitrary number or charactercharacter

01

2

Memory cells are numbered and accessing Memory cells are numbered and accessing any cell in memory takes unit time.any cell in memory takes unit time.

1111

Primitive OperationsPrimitive Operations Basic computations Basic computations

performed by an performed by an algorithmalgorithm

Identifiable in Identifiable in pseudocodepseudocode

Largely independent from Largely independent from the programming the programming languagelanguage

Exact definition not Exact definition not important (we will see important (we will see why later)why later)

Assumed to take a Assumed to take a constant amount of time constant amount of time in the RAM modelin the RAM model

Examples:Examples:• Assigning a value to a Assigning a value to a

variablevariable• Calling a methodCalling a method• Performing an arithmetic Performing an arithmetic

operation such as additionoperation such as addition• Comparing two numbersComparing two numbers• Indexing into an arrayIndexing into an array• Following an object Following an object

referencereference• Returning from a methodReturning from a method

1212

Counting Primitive Operations (§1.1)Counting Primitive Operations (§1.1) By inspecting the pseudocode, we can determine By inspecting the pseudocode, we can determine

the maximum number of primitive operations the maximum number of primitive operations executed by an algorithm, as a function of the input executed by an algorithm, as a function of the input sizesize

AlgorithmAlgorithm arrayMax(A,n); arrayMax(A,n); # operations# operationscurrentMaxcurrentMax AA[0][0] 2 2 [index into array; [index into array;

assign value]assign value]forfor ii 11 toto nn 1 1 dodo 11 n n [n comparisons; [n comparisons;

initialization]initialization]ifif AA[[ii] ] currentMaxcurrentMax thenthen 2(2(nn 1) 1) [n-1 loops-index; [n-1 loops-index;

compare]compare]currentMaxcurrentMax AA[[ii]] 2(2(nn 1) 1) [n-1 loops-index; [n-1 loops-index;

assign]assign]{ increment counter { increment counter ii } } 2(2(nn 1) 1) [n-1 loops- add; assign][n-1 loops- add; assign]returnreturn currentMaxcurrentMax 11

5n (at least)5n (at least) TotalTotal 7 7nn 2 (worst case) 2 (worst case)

1313

Estimating Running TimeEstimating Running Time

Algorithm Algorithm arrayMaxarrayMax executes executes 77nn 2 2 primitive primitive operations in the worst case. Define:operations in the worst case. Define:aa = Time taken by the fastest primitive operation= Time taken by the fastest primitive operation

bb = Time taken by the slowest primitive = Time taken by the slowest primitive operationoperation

Let Let TT((nn)) be worst-case time of be worst-case time of arrayMax.arrayMax. ThenThen

a a (7(7nn 2) 2) TT((nn)) bb(7(7nn 2) 2) Hence, the running time Hence, the running time TT((nn)) is bounded by is bounded by

two linear functionstwo linear functions

1414

Growth Rate of Running TimeGrowth Rate of Running Time

Changing the hardware/ software Changing the hardware/ software environment environment • Affects Affects TT((nn)) by a constant factor, but by a constant factor, but• Does not alter the growth rate of Does not alter the growth rate of TT((nn))

The linear growth rate of the The linear growth rate of the running time running time TT((nn)) is an intrinsic is an intrinsic property of the property of the algorithmalgorithm arrayMaxarrayMax

1515

Growth RatesGrowth Rates

Growth rates of Growth rates of functions:functions:• Linear Linear nn• Quadratic Quadratic nn22

• Cubic Cubic nn33

In a log-log chart, the In a log-log chart, the slope of the line slope of the line corresponds to the corresponds to the growth rate of the growth rate of the functionfunction

1E+01E+21E+41E+61E+8

1E+101E+121E+141E+161E+181E+201E+221E+241E+261E+281E+30

1E+0 1E+2 1E+4 1E+6 1E+8 1E+10n

T(n

)

Cubic

Quadratic

Linear

1616

Constant FactorsConstant Factors

The growth rate is not affected byThe growth rate is not affected by• constant factors or constant factors or • lower-order termslower-order terms

ExamplesExamples• 101022nn 101055 is a linear functionis a linear function• 101055nn22 10 1088nn is a quadratic functionis a quadratic function

1717

Big-Oh Notation (§1.2)Big-Oh Notation (§1.2) Important definition:Important definition: Given functions Given functions ff((nn) )

and and gg((nn)), we say that , we say that ff((nn) ) is is OO((gg((nn)))) if there if there are positive constants are positive constants cc and and nn00 such that such that

ff((nn)) cgcg((nn) ) for for n n nn00

OO((nn)) is read as big-oh of n. is read as big-oh of n.

Example: Example: Prove: Prove: 22nn 1010 is is OO((nn))

• We want to find c and We want to find c and nn00 such that such that 22nn ++ 1010 cn for n cn for n nn00

Realize the values found will not be unique!Realize the values found will not be unique!

1818

Prove: 2n + 10 is Prove: 2n + 10 is O(n)O(n) We want to find c and We want to find c and nn00 such that such that

22nn ++ 1010 cn for n cn for n nn00

The yellow work below is scratch work to find a c The yellow work below is scratch work to find a c and nand n0 0 that works. that works.

Start with 2n + 10 Start with 2n + 10 cn cn and try to solve for nand try to solve for n ((cc 2) 2) n n 10 10 n n 10/( 10/(cc 2) 2)

So, one possible choice (but not the only one) So, one possible choice (but not the only one) would be to let c = 3 and nwould be to let c = 3 and n00 = 10. = 10.

This is not handed in when a proof is requested.This is not handed in when a proof is requested. Now we can construct a proof!Now we can construct a proof!

1919

Prove: 2n + 10 is Prove: 2n + 10 is O(n)O(n) We want to find c and We want to find c and nn00 such that such that

22nn ++ 1010 cn for n cn for n nn00

• Proof: Pick Proof: Pick c c = 3 and = 3 and nn0 0 = 10. Then, for n = 10. Then, for n nn0 0 = = 10,10,

n n ≥ 10/1 = 10/(3-2) = 10/(c - 2)≥ 10/1 = 10/(3-2) = 10/(c - 2)

Thus, (c - 2) n ≥ 10 andThus, (c - 2) n ≥ 10 and

cn - 2n ≥ 10cn - 2n ≥ 10

or 2n + 10 ≤ cnor 2n + 10 ≤ cn

So, we have found c = 3 and nSo, we have found c = 3 and n00 = 10 such that = 10 such that

2n + 10 2n + 10 cn for n cn for n nn00

Consequently, 2n + 10 is O(n)

2020

Big-Oh ExampleBig-Oh Example

Example: the function Example: the function nn22 is not is not OO((nn))• We wantWe want

nn22 cncn n n cc

• The above inequality cannot be The above inequality cannot be satisfied since satisfied since cc must be a constant must be a constant

2121

More Big-Oh ExamplesMore Big-Oh Examples 7n-27n-2

Claim: 7n-2 is O(n)need c > 0 and n0 1 such that 7n-2 cn for n n0

this is true for c = 7 and n0 = 1

3n3 + 20n2 + 53n3 + 20n2 + 5 is O(n3)need c > 0 and n0 1 such that 3n3 + 20n2 + 5 cn3 for n

n0

this is true for c = 4 and n0 = 21 3 log n + log log n3 log n + log log n is O(log n)need c > 0 and n0 1 such that 3 log n + log log n clog n

for n n0


2222


Claim: 7n-2 is O(n)need c > 0 and n0 1 such that 7n-2 cn for n n0


3n3 + 20n2 + 53n3 + 20n2 + 5 is O(n3)need c > 0 and n0 1 such that 3n3 + 20n2 + 5 cn3 for n

n0

this is true for c = 4 and n0 = 21 3 log n + log log n3 log n + log log n is O(log n)need c > 0 and n0 1 such that 3 log n + log log n clog n

for n n0


2323


Claim: 7n-2 is O(n)Scratchwork: Need c > 0 and n0 1 such that

7n-2 cn for n n0

7n-2 ≤ cn

7n - cn ≤ 2 (7-c) n ≤ 2


2424

Big-Oh and Growth RateBig-Oh and Growth Rate The big-Oh notation gives an upper bound on The big-Oh notation gives an upper bound on

the growth rate of a functionthe growth rate of a function The statement “The statement “ff((nn) ) is is OO((gg((nn))))” means that the ” means that the

growth rate of growth rate of ff((nn) ) is no more than the growth is no more than the growth rate of rate of gg((nn))

We can use the big-Oh notation to rank We can use the big-Oh notation to rank functions according to their growth ratefunctions according to their growth rate

Assume:Assume: ff((nn) ) is is OO((gg((nn)))) gg((nn) ) is is OO((ff((nn))))

gg((nn) ) grows grows moremore

YesYes NoNo

ff((nn) ) grows moregrows more NoNo YesYes

Same growthSame growth YesYes YesYes

2525

Big-Oh RulesBig-Oh Rules If is If is ff((nn)) a polynomial of degree a polynomial of degree dd, then , then

ff((nn)) is is OO((nndd)), i.e.,, i.e.,1.1.Drop lower-order termsDrop lower-order terms

2.2.Drop constant factorsDrop constant factors Use the smallest possible class of Use the smallest possible class of

functionsfunctions• Say “Say “22nn is is OO((nn))”” instead of “instead of “22nn is is OO((nn22))””

Use the simplest expression of the classUse the simplest expression of the class• Say “Say “33nn 55 is is OO((nn))”” instead of “instead of “33nn 55 is is

OO(3(3nn))””

2626

Asymptotic Algorithm AnalysisAsymptotic Algorithm Analysis The asymptotic analysis of an algorithm determines The asymptotic analysis of an algorithm determines

the running time in big-Oh notationthe running time in big-Oh notation To perform the asymptotic analysisTo perform the asymptotic analysis

• We find the worst-case number of primitive operations We find the worst-case number of primitive operations executed as a function of the input sizeexecuted as a function of the input size

• We express this function with big-Oh notationWe express this function with big-Oh notation Example:Example:

• We determine that algorithm We determine that algorithm arrayMaxarrayMax executes at most executes at most 77nn 2 2 primitive operationsprimitive operations

• We say that algorithm We say that algorithm arrayMaxarrayMax “runs in “runs in OO((nn) ) time”time” Since constant factors and lower-order terms are Since constant factors and lower-order terms are

eventually dropped anyhow, we can disregard them eventually dropped anyhow, we can disregard them when counting primitive operations. This means a bit when counting primitive operations. This means a bit of slop is OK as long as you don’t mess up the counts of slop is OK as long as you don’t mess up the counts involving ninvolving n

2727

Computing Prefix AveragesComputing Prefix Averages

We further illustrate We further illustrate asymptotic analysis with asymptotic analysis with two algorithms for prefix two algorithms for prefix averagesaverages

The The ii-th prefix average -th prefix average of an array of an array XX is the is the average of the first average of the first ((ii 1) 1) elements of elements of XX::

AA[[ii]] XX[0] [0] XX[1] [1] … … XX[[ii])/(])/(ii+1)+1)

Computing the array Computing the array AA of of prefix averages of prefix averages of another array another array XX has has many applications.many applications.

0

5

10

15

20

25

30

35

1 2 3 4 5 6 7

X

A

2828

Computing Prefix AveragesComputing Prefix Averages

One application of the prefix average One application of the prefix average calculation:calculation:• Given the year-by-year returns on a stock, see Given the year-by-year returns on a stock, see

what the stock’s average annual returns were what the stock’s average annual returns were for last year, the last three years, the last ten for last year, the last three years, the last ten years, etc. years, etc.

As you can see by the previous graph, it As you can see by the previous graph, it has the effect of smoothing out the data has the effect of smoothing out the data especially when that data is varying a lot.especially when that data is varying a lot.

2929

Prefix Averages (Quadratic)Prefix Averages (Quadratic) The following algorithm computes prefix averages in The following algorithm computes prefix averages in

quadratic time by applying the definition rather directlyquadratic time by applying the definition rather directly

AlgorithmAlgorithm prefixAverages1prefixAverages1((X, nX, n))InputInput array array XX of of nn integers integersOutputOutput array array AA of prefix averages of of prefix averages of XX #operations#operations

AA new array of new array of nn integers integers forfor ii 00 toto nn 1 1 dodo nn

ss XX[0] [0] nnforfor jj 11 toto ii dodo 1 1 2 2 …… ( (nn 1) 1)

ss ss XX[[jj]] 1 1 2 2 …… ( (nn 1) 1)AA[[ii]] ss ( (ii 1) 1) nn

returnreturn A A 11

3030

Arithmetic ProgressionArithmetic Progression

The running time of The running time of prefixAverages1 prefixAverages1 isisOO(1 (1 2 2 ……nn))

The sum of the first The sum of the first nn integers is integers is nn((nn 1) 1) 22• There is a simple There is a simple

visual proof of this factvisual proof of this fact Thus, algorithm Thus, algorithm

prefixAverages1 prefixAverages1 runs in runs in OO((nn22) ) time time

Recall this sum is Recall this sum is Gauss’ formula that Gauss’ formula that was discussed in cpsc was discussed in cpsc 171!171!

0

1

2

3

4

5

6

7

1 2 3 4 5 6

3131

Prefix Averages (Linear)Prefix Averages (Linear) The following algorithm computes prefix The following algorithm computes prefix

averages in linear time by keeping a running sumaverages in linear time by keeping a running sum

AlgorithmAlgorithm prefixAverages2prefixAverages2((X, nX, n))InputInput array array XX of of nn integers integersOutputOutput array array AA of prefix averages of of prefix averages of XX #operations#operations

AA new array of new array of nn integers integersss 0 0 11forfor ii 00 toto nn 1 1 dodo nn

ss ss XX[[ii]] nnAA[[ii]] ss ( (ii 1) 1) nn

returnreturn A A 11 Algorithm Algorithm prefixAverages2 prefixAverages2 runs in runs in OO((nn) ) time time

3232

Summations (Sec. 1.3.1)Summations (Sec. 1.3.1) These often arise in analysis of algorithms These often arise in analysis of algorithms

because they arise when loops are examined.because they arise when loops are examined. Some of these are standard ones seen (but Some of these are standard ones seen (but

probably forgotten) in Algebra II in high school.probably forgotten) in Algebra II in high school.

One, of course, is Gauss’ formula.One, of course, is Gauss’ formula. Another is the Another is the geometric summationgeometric summation nni=0 i=0 ai = i = (1-a(1-an+1n+1)/(1-a))/(1-a)You should know the special case ofYou should know the special case of

1 + 2 + 4 + 8 +...+ 21 + 2 + 4 + 8 +...+ 2n-1n-1 = 2 = 2nn-1, the largest -1, the largest interger representable in n bits.interger representable in n bits.

Some math you will need Some math you will need

3333

Logarithms and Exponents (Sec. 1.3.2)Logarithms and Exponents (Sec. 1.3.2)

Definition: Definition: loglogbba = c of a = ba = c of a = bcc

properties of logarithms:properties of logarithms:loglogbb(xy) = log(xy) = logbbx + logx + logbbyyloglogbb (x/y) = log (x/y) = logbbx - logx - logbbyyloglogbbxxa a = alog= alogbbxxloglogbba = loga = logxxa/loga/logxxbb

properties of exponentialsproperties of exponentials::aa(b+c)(b+c) = a = abba a cc

aabcbc = (a = (abb))cc

aabb /a /acc = a = a(b-c)(b-c)

b = a b = a loglogaabb

bbcc = a = a cc loglogaabb

3434

Relatives of Big-OhRelatives of Big-Oh big-Omegabig-Omega

• f(n) is f(n) is (g(n)) if there is a constant c > 0 (g(n)) if there is a constant c > 0

and an integer constant nand an integer constant n00 1 such that 1 such that

f(n) f(n) cg(n) for n cg(n) for n n n00

Intuitively, this says up to a constant factor, f(n) Intuitively, this says up to a constant factor, f(n) asymptotically is greater than or equal to g(n)asymptotically is greater than or equal to g(n)

big-Thetabig-Theta• f(n) is f(n) is (g(n)) if there are constants c’ > 0 and c’’ > 0 (g(n)) if there are constants c’ > 0 and c’’ > 0

and an integer constant nand an integer constant n00 1 such that c’g(n) 1 such that c’g(n) f(n) f(n) c’’ c’’••g(n) for n g(n) for n n n00

Intuitively, this says up to a constant factor, f(n) and Intuitively, this says up to a constant factor, f(n) and g(n) are asymptotically the same.g(n) are asymptotically the same.

Note: This is the concept I introduced in cpsc 171.Note: This is the concept I introduced in cpsc 171.

3535

Relatives of Big-OhRelatives of Big-Oh

little-ohlittle-oh• f(n) is o(g(n)) if, for any constant c > 0, there is f(n) is o(g(n)) if, for any constant c > 0, there is

an integer constant nan integer constant n00 0 such that f(n) 0 such that f(n) cg(n) cg(n) for n for n n n00

Intuitively, this says f(n) is, up to a constant, Intuitively, this says f(n) is, up to a constant, asymptotically strictly less than g(n).asymptotically strictly less than g(n).

little-omegalittle-omega• f(n) is f(n) is (g(n)) if, for any constant c > 0, there is (g(n)) if, for any constant c > 0, there is

an integer constant nan integer constant n00 0 such that f(n) 0 such that f(n) cg(n) cg(n) for n for n n n00

Intuitively, this says f(n) is, up to a constant, Intuitively, this says f(n) is, up to a constant, asymptotically strictly greater than g(n).asymptotically strictly greater than g(n).

These are not used as much as the earlier These are not used as much as the earlier definitions, but they round out the picture.definitions, but they round out the picture.

3636

Summary for Intuition for Summary for Intuition for Asymptotic NotationAsymptotic Notation

Big-OhBig-Oh• f(n) is O(g(n)) if f(n) is asymptotically f(n) is O(g(n)) if f(n) is asymptotically less than or equalless than or equal to to

g(n)g(n)

big-Omegabig-Omega• f(n) is f(n) is (g(n)) if f(n) is asymptotically (g(n)) if f(n) is asymptotically greater than or equalgreater than or equal to to

g(n)g(n)big-Thetabig-Theta• f(n) is f(n) is (g(n)) if f(n) is asymptotically (g(n)) if f(n) is asymptotically equalequal to g(n) to g(n)

little-ohlittle-oh• f(n) is o(g(n)) if f(n) is asymptotically f(n) is o(g(n)) if f(n) is asymptotically strictly lessstrictly less than g(n) than g(n)

little-omegalittle-omega• f(n) is f(n) is (g(n)) if is asymptotically (g(n)) if is asymptotically strictly greaterstrictly greater than g(n) than g(n)

3737

Example Uses of the Example Uses of the Relatives of Big-OhRelatives of Big-Oh

f(n) is (g(n)) if, for any constant c > 0, there is an integer constant n0 0 such that f(n) cg(n) for n n0

need 5n02 c•n0 given c, the n0 that satisfies this is n0 c/5 0

5n2 is (n)

f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0 1 such that f(n) cg(n) for n n0

let c = 1 and n0 = 1

5n2 is (n)

f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0 1 such that f(n) cg(n) for n n0

let c = 5 and n0 = 1

5n2 is (n2)

3838

A MORE PRECISE DEFINITION OF O, A MORE PRECISE DEFINITION OF O, (only for those with calculus backgrounds)(only for those with calculus backgrounds)

Definition: Let f and g be functions defined on the positive real numbers with real values.

We say g is in O(f) if and only if

lim g(n)/f(n) = c

n -> for some nonnegative real number c--- i.e. the limit exists and is not infinite.

We say f is in (g) if and only if

f is in O(g) and g is in O(f)

Note: Often to calculate these limits you need L'Hopital's Rule.

3939

Why Asymptotic Behavior is ImportantWhy Asymptotic Behavior is Important

1) Allows us to compare two algorithms 1) Allows us to compare two algorithms with respect to running time on with respect to running time on large data large data sets.sets.

2) Helps us understand the maximum size 2) Helps us understand the maximum size of input that can be handled in a given of input that can be handled in a given time, provided we know the environment time, provided we know the environment in which we are running.in which we are running.

3) Stresses the fact that even dramatic 3) Stresses the fact that even dramatic speedups in hardware do not overcome speedups in hardware do not overcome the handicap of an asymtotically slow the handicap of an asymtotically slow algorithm.algorithm.

4040

ORDER WINS OUTORDER WINS OUTThe TRS-80

Main language support: BASIC - typically a slow running language

For more details on TRS-80 see:

http://mate.kjsl.com/trs80/

http://ds.dial.pipex.com/town/park/abm64/CrayWWWStuff/Cfaqp1.html#TOC3

The CRAY-YMP

Language used in example: FORTRAN- a fast running language

For more details on CRAY-YMP see:

4141

CRAY YMP TRS-80with FORTRAN with BASICcomplexity is 3n3 complexity is 19,500,000n

n is:

10

100

1000

2500

10000

1000000

3 microsec 200 millisec

3 millisec 2 sec

3 sec 20 sec

50 sec 50 sec

49 min 3.2 min

95 years 5.4 hours

microsecond (abbr microsecond (abbr µµsec)sec) One-millionth of a second. One-millionth of a second.millisecond (abbr msec)millisecond (abbr msec) One-thousandth of a second. One-thousandth of a second.

4242

Proof techniques (Sec. 1.3.3)Proof techniques (Sec. 1.3.3)

By exampleBy example• You often hear profs say “you can’t prove by You often hear profs say “you can’t prove by

an example”. an example”. • That is true unless you have some very special That is true unless you have some very special

situations.situations. Situation 1: Prove that there is an element Situation 1: Prove that there is an element

x in a set S that has property P.x in a set S that has property P.• To prove this, you need toTo prove this, you need to

Define one element xDefine one element x Show x is in the set SShow x is in the set S Show x has property PShow x has property P

4343


Situation 2: Show that the claim “Every element x Situation 2: Show that the claim “Every element x in a set S has property P” is false.in a set S has property P” is false.• Now all you have to do is produce one Now all you have to do is produce one counterexample counterexample

to the statement.to the statement.• i.e. define an element x i.e. define an element x • Show x is in the set SShow x is in the set S• Show x does NOT have property PShow x does NOT have property P

Example:Example: Prove that the statement every number Prove that the statement every number of the form 2of the form 2ii – 1, for i >= 1, is prime is false. – 1, for i >= 1, is prime is false.• Let i= 4. Then 2Let i= 4. Then 244 -1 = 15 = (3)(5) is not prime. -1 = 15 = (3)(5) is not prime.• Note- the choice of i is not unique. You need only find Note- the choice of i is not unique. You need only find

one i which makes the statement false.one i which makes the statement false.

4444

Proof techniques (Sec. 1.3.3)Proof techniques (Sec. 1.3.3) Many theorems are of the formMany theorems are of the form

• If P, then Q.If P, then Q.

There are several approaches for proving this.There are several approaches for proving this. A A direct proofdirect proof does the following: does the following:

• Assume P is true.Assume P is true.• Using that information, deduce using known facts Using that information, deduce using known facts

that Q is true.that Q is true.• Students often wonder why you assume P is true.Students often wonder why you assume P is true.

Logic dictates the following truth table:Logic dictates the following truth table:P Q P P Q P Q QT T TT T T The last two entries often seem The last two entries often seem T F F strange to students. See class demoT F F strange to students. See class demoF T T which shows why those entries areF T T which shows why those entries areF F T specified.F F T specified.

4545

Proof techniques (Sec. 1.3.3)Proof techniques (Sec. 1.3.3) A contrapositive proof does the following:A contrapositive proof does the following:

• The contrapositive of “If P, then Q” isThe contrapositive of “If P, then Q” is

If not Q, then not PIf not Q, then not P• The contrapositive is logically equivalent to the The contrapositive is logically equivalent to the

original, i.e. both have the same truth table:original, i.e. both have the same truth table:

P Q not Q not P not Q -> not PP Q not Q not P not Q -> not P

T TT T F F F F TT

T FT F T F T F F F

F TF T F T F T TT

F FF F T T T T TT• Now, assume not Q and deduce not P.Now, assume not Q and deduce not P.

4646


A proof by contradiction does the A proof by contradiction does the following:following:• Assume P is true and Q is false.Assume P is true and Q is false.• With these assumptions, try to reach a With these assumptions, try to reach a

contradiction--- i.e. logically deduce some contradiction--- i.e. logically deduce some statement R and the statement not R.statement R and the statement not R.

• As we would always assume P is true when As we would always assume P is true when proving “If P, then Q”, then our assumption proving “If P, then Q”, then our assumption that Q is false must be wrong --- i.e. Q must be that Q is false must be wrong --- i.e. Q must be truetrue

4747


Which is approach is best?Which is approach is best?• Direct proofDirect proof• Contrapositive proofContrapositive proof• Proof by contradictionProof by contradiction

None of these- all are logically equivalent.None of these- all are logically equivalent. Which you choose depends upon the way you see Which you choose depends upon the way you see

the proof.the proof. Remember, a proof just tries to show others what Remember, a proof just tries to show others what

(acceptable) reasoning you used to draw a (acceptable) reasoning you used to draw a conclusion. Therefore, any of these approaches conclusion. Therefore, any of these approaches are fine.are fine.

4848


The age old question is, “What can I The age old question is, “What can I assume is true when I am doing a proof?”assume is true when I am doing a proof?”

Normally, you don’t start from Peano’s Normally, you don’t start from Peano’s axioms for the integers and build axioms for the integers and build everything else from those (although, that everything else from those (although, that is an interesting approach to try sometime is an interesting approach to try sometime in your lifetime!)in your lifetime!)

You need to use some judgment as to You need to use some judgment as to what the course is assuming and what I what the course is assuming and what I am asking you to prove.am asking you to prove.

4949

Proof techniques (Sec. 1.3.3)Proof techniques (Sec. 1.3.3) Many analyzes of complexity involve Many analyzes of complexity involve

statements of the form statements of the form • Q(n) is true for all n ≥ 1 (or for all n ≥ k, for Q(n) is true for all n ≥ 1 (or for all n ≥ k, for

some k where n is an integer.some k where n is an integer.• This is a claim about an infinite set of numbers.This is a claim about an infinite set of numbers.• When the integers are defined using Peano’s When the integers are defined using Peano’s

axioms, one of the axioms is the Induction axioms, one of the axioms is the Induction Principle which is assumed in one of two Principle which is assumed in one of two equivalent forms:equivalent forms:

5050

Induction PrincipleInduction Principle

Let Q(n) be a statement about integers n and we Let Q(n) be a statement about integers n and we are to prove that “Q(n) is true for all integers n ≥ are to prove that “Q(n) is true for all integers n ≥ k, where k is an integer.”k, where k is an integer.”

One form of the Induction Principle says the One form of the Induction Principle says the above is true if we can prove the following:above is true if we can prove the following:• 1) Q(k) is true1) Q(k) is true• 2) If Q(n) is true for i < n, then Q(n) is true.2) If Q(n) is true for i < n, then Q(n) is true.

An alternate form replaces (2) withAn alternate form replaces (2) with• 2) If Q(n) is true, then Q(n + 1) is true.2) If Q(n) is true, then Q(n + 1) is true.

These forms are logically equivalent even though These forms are logically equivalent even though the first form is called (unfortunately) the first form is called (unfortunately) strong strong inductioninduction and the second form is called and the second form is called weak weak induction.induction.

5151

Induction and Proofs Involving RecursionInduction and Proofs Involving Recursion Many algorithms use recursion instead of Many algorithms use recursion instead of

iteration for control.iteration for control. Analyzing these algorithms often entail a timing Analyzing these algorithms often entail a timing

function that is expressed as a function that is expressed as a recurrence recurrence relationrelation..

Example: (A recursive solution for finding the Example: (A recursive solution for finding the maximum number in an array)maximum number in an array)• Details are on page 12, but the basic step isDetails are on page 12, but the basic step is

if n = 1 then if n = 1 then

return A[0]return A[0]

else return max {recursiveMax(A,n-1), A[n-1]}else return max {recursiveMax(A,n-1), A[n-1]}

where max is the maximum function for two elements.where max is the maximum function for two elements.

5252

Analyze the recursiveMax algorithmAnalyze the recursiveMax algorithm

Count every comparison, array reference, Count every comparison, array reference, recursive call, max calculation, or return as a recursive call, max calculation, or return as a single primitive.single primitive.

T(n) = 3 if n = 1 T(n) = 3 if n = 1 [check, access, return][check, access, return]T(n-1) + 7 otherwise T(n-1) + 7 otherwise

[T(n-1) is the timing for the recursive call;[T(n-1) is the timing for the recursive call;check n<> 1, access A[n-1], compute max (4 check n<> 1, access A[n-1], compute max (4

operations- if a < b return b else return a), operations- if a < b return b else return a), return ]return ]

This formula is called a This formula is called a recurrence relationrecurrence relation..

5353

Induction and Proofs Involving RecursionInduction and Proofs Involving Recursion

Often we want a Often we want a closed form (i.e. an explicit closed form (i.e. an explicit formula)formula) for the recurrence relation. for the recurrence relation.

Finding the closed form MAY be difficult.Finding the closed form MAY be difficult. But, after finding a possible candidate, we often But, after finding a possible candidate, we often

have to prove it is correct.have to prove it is correct. For example, we claim that the recurrence For example, we claim that the recurrence

relation derived for recursiveMax is the closed relation derived for recursiveMax is the closed form T(n) = 7n -4.form T(n) = 7n -4.

To show this is correct, an induction proof is often To show this is correct, an induction proof is often used.used.

5454

Proof techniques (Sec. 1.3.3)Proof techniques (Sec. 1.3.3) Loop Invariants- Loop Invariants- a technique used to prove a technique used to prove

statements about loops.statements about loops. To prove some statement S about a loop, define S in To prove some statement S about a loop, define S in

terms of a series of simpler statements Sterms of a series of simpler statements S00, S, S11, ..., S, ..., Skk wherewhere• 1) The initial claim S1) The initial claim S0 0 is true before the loop is true before the loop

beginsbegins• 2) If S2) If Si-1 i-1 is true before iteration i begins, then one is true before iteration i begins, then one

can show that Scan show that Sii is true after iteration is is over is true after iteration is is over• 3) The final statement, S3) The final statement, Skk, implies the statement S , implies the statement S

is true.is true. Use this technique to prove that the arrayMax Use this technique to prove that the arrayMax

algorithm is correct.algorithm is correct.

5555

Basic probability (Sec. 1.3.4)Basic probability (Sec. 1.3.4)

We will introduce these concepts later as We will introduce these concepts later as needed.needed.

They are used predominately when we use They are used predominately when we use random data to analyze average cases.random data to analyze average cases.

The average case analysis is almost The average case analysis is almost always much more difficult than the worst always much more difficult than the worst case or best case analysis. case or best case analysis. • Some algorithms have not had there average Some algorithms have not had there average

cases analyzed yet--- an example is an cases analyzed yet--- an example is an interesting sort called the Shell Sort.interesting sort called the Shell Sort.

5656

Amortization Techniques (Sect. 1.5)Amortization Techniques (Sect. 1.5)

We will postpone these, also, for a bit We will postpone these, also, for a bit later.later.

They are just now coming into use in They are just now coming into use in computer science.computer science.

The techniques come from financial The techniques come from financial analysis and provide a way of performing analysis and provide a way of performing average-case analysis without using average-case analysis without using probability theory.probability theory.

5757

Experimentation (Sect 1.6)Experimentation (Sect 1.6) Asymptotic analysis is powerful, but does have Asymptotic analysis is powerful, but does have

some limitations.some limitations. It does not provide insight into the constant It does not provide insight into the constant

factors hiding behind the big-oh notation.factors hiding behind the big-oh notation. Another problem is that most analysis deals with Another problem is that most analysis deals with

the worst case, as the average case is often the worst case, as the average case is often unbelievably difficult to analyze.unbelievably difficult to analyze.

There may be other questions we wish to answer There may be other questions we wish to answer about the algorithm other than what its worst about the algorithm other than what its worst case is.case is.

Some algorithms are just too complicated to even Some algorithms are just too complicated to even find bounds for their performance.find bounds for their performance.

5858

First, focus on what question you want to First, focus on what question you want to answeranswer

Do we want to estimate the asymptotic running Do we want to estimate the asymptotic running time for the average case?time for the average case?

Do we wish to see which of two algorithms is Do we wish to see which of two algorithms is faster for a given range of values?faster for a given range of values?

If an algorithm has constants that affect its If an algorithm has constants that affect its behavior, can we determine which ones yield the behavior, can we determine which ones yield the best performance?best performance?

If a function is supposed to minimize or maximize If a function is supposed to minimize or maximize its input, can we see how close it comes to its its input, can we see how close it comes to its optimal value?optimal value?

You may choose different experiments depending You may choose different experiments depending on the question.on the question.

5959

Second, decide what you want to measureSecond, decide what you want to measure

Running time is often one of the poorest Running time is often one of the poorest choices as it is affected by so many choices as it is affected by so many factors. factors. • However, this is a useful measure in some However, this is a useful measure in some

cases.cases. Other possibilities:Other possibilities:

• Count memory referencesCount memory references• Count comparisonsCount comparisons• Count arithmetic operationsCount arithmetic operations• Count any operation (or operations) that seem Count any operation (or operations) that seem

to contribute to the work of the algorithm.to contribute to the work of the algorithm.

6060

Third, use good practices in choosing test Third, use good practices in choosing test datadata

You want data that covers your possible You want data that covers your possible input space well.input space well.

Do not assume random data is what you Do not assume random data is what you always need for average data.always need for average data.

You may need to analyze your input space You may need to analyze your input space and determine the probability of certain and determine the probability of certain kinds of data occuring.kinds of data occuring.• Ex. How likely is it that quicksort will encounter Ex. How likely is it that quicksort will encounter

already sorted data?already sorted data?

6161

Fourth, code the algorithm(s) and record Fourth, code the algorithm(s) and record environmental dataenvironmental data

A common problem with experimental A common problem with experimental data is a poor implementation.data is a poor implementation.• Ex. Quicksort Ex. Quicksort

You need to record carefully the You need to record carefully the environmental data:environmental data:• CPU type and speedCPU type and speed• Number of CPUsNumber of CPUs• Memory size (including main and cache)Memory size (including main and cache)• Disk speed (if external input from disk is used)Disk speed (if external input from disk is used)• Memory bus speedMemory bus speed

6262

Fifth, present the resulting data in a useful Fifth, present the resulting data in a useful manner--- i.e. perform good data analysis manner--- i.e. perform good data analysis

and visualizationand visualization A basic principle: Viewing data in tables is not as A basic principle: Viewing data in tables is not as

useful as showing graphical plots.useful as showing graphical plots. Two useful methods are:Two useful methods are:

• The The ratio testratio test – – Tries to derive a function f(n) = nTries to derive a function f(n) = ncc for the main term in the for the main term in the

algorithm’s running time, for some constant c > 0.algorithm’s running time, for some constant c > 0. Only useful for polynomial running times.Only useful for polynomial running times.

• The The power test power test –– Can produce a good estimate for the running time t(n) of Can produce a good estimate for the running time t(n) of

an algorithm without producing a good guess for that an algorithm without producing a good guess for that running time in advance.running time in advance.

Both these tests can at best achieve an accurate Both these tests can at best achieve an accurate c in the range [c=0.5, c+0.5], but no better. Such c in the range [c=0.5, c+0.5], but no better. Such a range, however, is not guaranteed.a range, however, is not guaranteed.

6363

The Ratio TestThe Ratio Test

Derive a function f(n) = nDerive a function f(n) = ncc, for c > 0, for , for c > 0, for the main term in the algorithm.the main term in the algorithm.

We will test experimentally whether the We will test experimentally whether the algorithm is algorithm is ΘΘ(n(ncc) or not.) or not.

Let t(n) be the actual running time of the Let t(n) be the actual running time of the algorithm on a specific problem instance algorithm on a specific problem instance of size n. (You might want this to be the of size n. (You might want this to be the average over several different types of average over several different types of data of size n).data of size n).

Plot the ratio r(n) = t(n)/f(n).Plot the ratio r(n) = t(n)/f(n).

6464

The Ratio Test- plotting r(n) = t(n)/f(n)The Ratio Test- plotting r(n) = t(n)/f(n)

If r(n) increases as n increases, thenIf r(n) increases as n increases, then• f(n) underestimates the running time t(n)f(n) underestimates the running time t(n)

If r(n) converges to 0, thenIf r(n) converges to 0, then• f(n) is an overestimate.f(n) is an overestimate.

If r(n) converges to some constant b >0, If r(n) converges to some constant b >0, thenthen• we have a good estimate for the growth rate of we have a good estimate for the growth rate of

t(n),t(n),• Moreover, the constant b is a good estimate for Moreover, the constant b is a good estimate for

the constant factor in the running time t(n).the constant factor in the running time t(n). See example.See example.

6565

The Ratio Test- plotting r(n) = t(n)/f(n)The Ratio Test- plotting r(n) = t(n)/f(n)ProblemsProblems

Can mislead you- for example, may look like it is Can mislead you- for example, may look like it is converging to 0, but with small data sets, you converging to 0, but with small data sets, you may not see this.may not see this.

In the selection sort example, working with nIn the selection sort example, working with n22, the , the constant is ½, but on a small data set, it looked constant is ½, but on a small data set, it looked as if convergence was to 0as if convergence was to 0• Recall- even with the best possibility, a 0.5 error is Recall- even with the best possibility, a 0.5 error is

possible.possible. We can rule out We can rule out ΘΘ(n) and (n) and ΘΘ(n(n33) as possibilities, ) as possibilities,

however.however. With a non-polynomial algorithm, all you will get With a non-polynomial algorithm, all you will get

is an upper bound.is an upper bound.• Is it useful to know that quicksort is not Is it useful to know that quicksort is not ΘΘ(n), but could (n), but could

be be ΘΘ(n(n22) ?) ?

6666

The Power TestThe Power Test

With this approach, we don’t need to make an With this approach, we don’t need to make an estimate in advance of the complexity.estimate in advance of the complexity.

Given a collection of data points (x,y) where x is Given a collection of data points (x,y) where x is the input size and y is t(x) from experimental the input size and y is t(x) from experimental runs, plot (log x, log y).runs, plot (log x, log y).

If t(n) = bnIf t(n) = bncc and (log x, log y) is (x’,y’), then y’ = and (log x, log y) is (x’,y’), then y’ = cx’ + b.cx’ + b.

So, if we plot on log-log paper, we can roughly So, if we plot on log-log paper, we can roughly determine b and c, if the plot is a straight line.determine b and c, if the plot is a straight line.

If the pairs grow without appearing to be a line, If the pairs grow without appearing to be a line, then t(n) is most likely super-polynomial.then t(n) is most likely super-polynomial.

If the pairs converge to a constant, then t(n) is If the pairs converge to a constant, then t(n) is most likely sublinear.most likely sublinear.

6767

ExampleExample

1

10

100

1,000

10,000

100,000

1,000,000

1 10 100 1,000n

n 2̂

100n

10n

n

6868

Cautions on Using Experimental AnalysisCautions on Using Experimental Analysis

You must choose good input test cases.You must choose good input test cases. For a given value n, it is often wise to average For a given value n, it is often wise to average

over different types of data.over different types of data. You must plot enough points to see if something You must plot enough points to see if something

is converging or not.is converging or not. Even with the best of data, the accuracy is not Even with the best of data, the accuracy is not

exact.exact. Nevertheless, the ratio test and the power test Nevertheless, the ratio test and the power test

are better than trying statistically to fit a are better than trying statistically to fit a polynomial to data points by regression methods polynomial to data points by regression methods which are very sensitive to noise.which are very sensitive to noise.

chapter 1

Documents