cs 221 analysis of algorithms instructor: don mclaughlin

CS 221

Analysis of AlgorithmsInstructor: Don McLaughlin

Theoretical Analysis of Algorithms

Recursive Algorithms One strategy for algorithmic design is

solve the problem using the same design but with a smaller problem The algorithm must be able use itself to

compute a solution but with subset of the problem

Must be one case that can be solved with evoking the whole algorithm again – Base Case

This is called recursion


Recursive Algorithms Recursion is elegant But is it efficient?


A recursive Max algorithm

Algorithm recursiveMax(A,n)Input: Array A storing n>=1 integersOutput: Maximum value in Aif n = 1 then

return A[0]return max{recursiveMax(A,n-1),A[n-1]


Efficiency –

T(n) = {3 if n = 1, T(n-1)+7 if otherwise}

or

T(n) = 7(n-1)+3 = 7n-2

* We will come back to this


Best-case Worst-case Average-case

A number if issues here Not as straight forward as you would

think


Average-case Sometimes Average Case is needed Best or Worst case may not be the

best representation of the typical case with “usual” input

Best or worst case may be rare Consider

A sort that typically gets a


Average-case Consider

A sort that typically gets a partially sorted list A concatenation of sorted sublists

A search that must find a match in a list …and the list is partially sorted


Average-case Can you take the average of best-case

and worst case?


Average-case Average case must consider the probability

of each case or a range of n N may not be arbitrary One strategy –

Divide range of n into classes Determine the probability that any given

n is from each respective class Use n in the analysis based on probability

distribution of each class


Asymptotic notation Remember that our analysis is

concerned with the efficiency of algorithms

so we determine a function that describes (to a reasonable degree) the run-time cost (T) of the algorithm

we are particularly interested in the growth of the algorithm’s cost as the size of the problem grows (n)


Asymptotic notation Often we need to know that run-time

cost (T) is broader terms …within certain boundaries We want to describe the efficiency of an

algorithm in terms of its asymptotic behavior


Big 0 Suppose we have a function of n

g(n) that we suggest gives us an upper

bound on the worst case behavior of our algorithm’s runtime –

which we have determined to be f(n) then…


Big 0 We describe the upper bound on the

growth of our run-time function f(n) –

f(n) is O(g(n)) f(n) is bounded from above by g(n) for all

significant values of n if

f(n) <= cg(n) for all n >= n0

there exists some constant c such that for all values of n >= n0

f(n) <= cg(n)


Big 0

from: Levitin, Anany, The Design and Analysis of Algorithms, Addison-Wesley, 2007


Big 0 …but be careful f(n) = O(g(n)) is incorrect the proper term is

f(n) is O(g(n))

to be absolutely correct f(n) Є O(g(n))


Big Ω Big Omega our function is bounded from below by

g(n) that is, f(n) is Ω(g(n))

if there exists some positive constant c such that f(n) >= cg(n), n >= n0

what does this mean?


Big Ω

from: Levitin, Anany, The Design and Analysis of Algorithms, Addison-Wesley, 2007


Big Θ Big Theta our function is bounded from above

and below by g(n) that is, f(n) is Θ(g(n))

if there exists two positive constants c1 and c2

such that c2g(n) <= f(n) >= c1g(n) for all n >= n0



Big Θ


Or said in another way O(g(n)): class of functions f(n) that grow

no faster than g(n)

Θ (g(n)): class of functions f(n) that grow at same rate as g(n)

Ω(g(n)): class of functions f(n) that grow at least as fast as g(n)


Little o o f(n) is o(g(n)) if f(n) < cg(n) for some n

>= n0 for all constants c >0


f(n) is asymptotically smaller than g(n)


Little ω ω

f(n) is ω(g(n)) if f(n) < cg(n) for all constants c and n>n0


f(n) is asymptotically larger than g(n)


Simplifying things a bit

Usually only really interested in the order of growth of our algorithm’s run-time function

Theoretical Analysis of Algorithms Suppose we have a run-time function like

T(n) = 4n3 + 12n2 + 2n + 7

So, to simplify our run-time function T, we can eliminate constant terms that are not

coefficients of n 4n3 + 12n2 + 2n

eliminate lowest order terms 4n3 + 12n2

Maybe only keep the highest order term 4n3

…and drop the coefficent of that term n3


So, T(n) = 4n3 + 12n2 + 2n + 7

therefore T(n) is O(n3)

or is it that T(n) is O(n6) (true or false)

True but it is considered bad form why?

Classes of Algorithmic Efficiency

Class Name Algorithms

1 Constant Time

Runs in constant time regardless of the size of the problem (n) Algorithms like this are rare

Log n Logarithmic Algorithms that pare away part of the problem by constant factor in each iteration

n Linear Algorithm’s T grows in linear proportion to growth of n

nlog n n-log-n Divide and conquer algorithms, often seen in recursive algorithms

n2 Quadratic Seen in algorithms that two level nested loops

n3 cubic Often seen in algorithms that three levels of nested loops, linear algebra

2n exponential Algorithms that grow as power of 2 – all possible subsets of a set

n! factorial All permutations of a set

based on: Levitin, Anany, The Design and Analysis of Algorithms, Addison-Wesley, 2007

Properties of Asymptotic Comparisons Transitivity

f(n) is O(g(n)) and g(n) is O(h(n)) then f(n) is O(h(n)) f(n) is Θ(g(n)) and g(n) is Θ(h(n)) then f(n) is Θ(h(n)) f(n) is Ω(g(n)) and g(n) is Ω(h(n)) then f(n) is Ω(h(n)) f(n) is o(g(n)) and g(n) is o(h(n)) then f(n) is o(h(n)) f(n) is ω(g(n)) and g(n) is ω(h(n)) then f(n) is ω(h(n))

Reflexivity f(n) is Θ(f(n)) f(n) is O(f(n)) f(n) is Ω(f(n))

Properties of Asymptotic Comparisons Symmetry

f(n) is Θ(g(n)) if and only if g(n) is Θ(f(n))

Tranpose Symmetry f(n) is O(g(n)) if and only if g(n) is Ω(f(n)) f(n) is o(g(n)) if and only if g(n) is ω(f(n))

Some rules of Asymptotic Notation d(n) is O(f(n)), then ad(n) is O(f(n)) for any

constant a > 0

d(n) is O(f(n)) and e(n) is O(g(n)), then d(n)+e(n) is O(f(n) + g(n))

d(n) is O(f(n)) and e(n) is O(g(n)), then d(n)e(n) is O(f(n)g(n))

d(n) is O(f(n)) and f(n) is O(g(n)), then d(n) is O(g(n))

Some rules of Asymptotic Notation f(n) is polynomial of degree d, the f(n) is O(nd) nx is O(an) for any fixed x > 0 and a > 1 lognx is O(log n) for any x > 0 logxn is O(ny) for any fixed constants x > 0

and y > 0

Homework

The Scream, by Edvard Munch, http://www.ibiblio.org/wm/paint/auth/munch/

Homework – due Wednesday, Sept. 3rd

Assume Various algorithms would run 100

primitive instructions per input element (n) (constant coefficient of 100)

…and it will be implemented on a processor that run 2 billion primitive instructions per second

then estimate the execution times for the following algorithmic efficiency classes

Homeworkwith n of

Log n n nlog n n2 n3 2n n!

10

50

200

1000

20000

100000

give answer in largest meaningful time increment

cs 221 analysis of algorithms instructor: don mclaughlin

Documents

storing n

given n

function of n gn

caseaverage case

typical case

worst case behavior

respective classuse

casesometimes average