alg efficiency

8/6/2019 Alg Efficiency

1/70

Analysis of Algorithm Efficiency

Dr. Yingwu Zhu


2/70

Measure Algorithm Efficiency

Time efficiency

How fast the algorithm runs; amount of time

required to accomplish the task

Our focus!

Space efficiency

Amount of memory required

Deals with the extra space the algorithm requires


3/70

Time Efficiency

Time efficiency depends on :

size of input

speed of machine

quality of source code

quality of compiler

These vary from one

platform to another

So, we cannot express time efficiency meaningfully in real time

units such as seconds!


4/70

Empirical analysis of time efficiency

Select a specific (typical) sample of inputs

Use physical unit of time (e.g., milliseconds)

orCount actual number of basic operations

executions

Analyze the empirical data Limitation: results dependent on the

particular computer and the sample of inputs


5/70

Theoretical analysis of time efficiency

Time efficiency is analyzed by determining the numberof repetitions of the basic operation as a function of

input size

Basic operation: the operation that contributes most towards

the running time of the algorithm

T(n) copC(n)

running timeexecution time

for basic

operation

Number of times

basic operation is

executed

Input size


6/70

Input size and basic operation examples

Problem Input size measure Basic operation

Searching for key in a

list ofn items

Number of lists items, i.e.

n

Key comparison

Multiplication of two

matrices

Matrix dimensions or total

number of elements

Multiplication of two

numbers

Checking primality of a

given integer n

nsize = number of digits

(in binary representation) Division

Typical graph problem #vertices and/or edgesVisiting a vertex or

traversing an edge


7/70

Time Efficiency

T(n) = (approximated by) number of times the basic

operation is executed.

Not only depends on the input size n, but also

depends on the arrangement of the input items Best case: not informative

Average case: difficult to determine

Worst case: is used to measure an algorithms

performance


8/70

Example: Sequential Search

Best case T(n) Worst case T(n)

Average case T(n)

Assume success search probability ofp


9/70

Order of Growth

Established framework for analyzing T(n)

Order of growth as n

Highest-order term is what counts

Remember, we are doing asymptotic analysis

As the input size grows larger it is the high order term

that dominates


10/70

Asymptotic Order of Growth

A way of comparing functions that ignores constantfactors and small input sizes

O(g(n)): class of functionsf(n) that grow no faster than

g(n)

(g(n)): class of functionsf(n) that grow at same rate as

g(n)

(g(n)): class of functionsf(n) that grow at least as fast

as g(n)


11/70

Big-oh


12/70

Big-omega


13/70

Big-theta


14/70

Upper Bound Notation

In general a function

f(n) is O(g(n)) if there exist positive constants c

and n0 such thatf(n) c g(n) for all n n0

Order of growth off(n) order of growth ofg(n)

(within constant multiple)

Formally

O(g(n)) = { f(n): positive constants c and n0 suchthat f(n) c g(n) n n0


15/70

Big-Oh

Examples

10n is O(n2)

5n+20 is O(n)


16/70

An Example: Insertion Sort

InsertionSort(A, n) {

for i = 2 to n {

key = A[i]

j = i - 1;

while (j > 0) and (A[j] > key) {A[j+1] = A[j]

j = j - 1

}

A[j+1] = key}

}


17/70



for i = 2 to n {

key = A[i]

j = i - 1;

while (j > 0) and (A[j] > key) {

A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

30 10 40 20

1 2 3 4

i = j = key = A[j] = A[j+1] =


18/70



for i = 2 to n {

key = A[i]

j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

30 10 40 20

1 2 3 4

i = 2 j = 1 key = 10A[j] = 30 A[j+1] = 10


19/70



for i = 2 to n {

key = A[i]

j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

30 30 40 20

1 2 3 4

i = 2 j = 1 key = 10A[j] = 30 A[j+1] = 30


20/70



for i = 2 to n {

key = A[i]

j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

30 30 40 20

1 2 3 4

i = 2 j = 1 key = 10A[j] = 30 A[j+1] = 30


21/70



for i = 2 to n {

key = A[i]

j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

30 30 40 20

1 2 3 4

i = 2 j = 0 key = 10A[j] = A[j+1] = 30


22/70



for i = 2 to n {

key = A[i]

j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

30 30 40 20

1 2 3 4

i = 2 j = 0 key = 10A[j] = A[j+1] = 30


23/70


24/70



for i = 2 to n {

key = A[i]

j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

10 30 40 20

1 2 3 4

i = 3 j = 0 key = 10A[j] = A[j+1] = 10


25/70



for i = 2 to n {

key = A[i]

j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

10 30 40 20

1 2 3 4

i = 3 j = 0 key = 40A[j] = A[j+1] = 10


26/70



for i = 2 to n {

key = A[i]

j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

10 30 40 20

1 2 3 4

i = 3 j = 0 key = 40A[j] = A[j+1] = 10


27/70



for i = 2 to n {

key = A[i]

j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

10 30 40 20

1 2 3 4

i = 3 j = 2 key = 40A[j] = 30 A[j+1] = 40


28/70



for i = 2 to n {

key = A[i]

j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

10 30 40 20

1 2 3 4

i = 3 j = 2 key = 40A[j] = 30 A[j+1] = 40


29/70



for i = 2 to n {

key = A[i]

j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

10 30 40 20

1 2 3 4

i = 3 j = 2 key = 40A[j] = 30 A[j+1] = 40


30/70



for i = 2 to n {

key = A[i]

j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

10 30 40 20

1 2 3 4

i = 4 j = 2 key = 40A[j] = 30 A[j+1] = 40


31/70



for i = 2 to n {

key = A[i]j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

10 30 40 20

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40


32/70



for i = 2 to n {

key = A[i]j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

10 30 40 20

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40


33/70



for i = 2 to n {

key = A[i]j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

10 30 40 20

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 20


34/70



for i = 2 to n {

key = A[i]j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

10 30 40 20

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 20


35/70



for i = 2 to n {

key = A[i]j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

10 30 40 40

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 40


36/70



for i = 2 to n {

key = A[i]j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

10 30 40 40

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 40


37/70



for i = 2 to n {

key = A[i]j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

10 30 40 40

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 40


38/70



for i = 2 to n {

key = A[i]j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

10 30 40 40

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40


39/70



for i = 2 to n {

key = A[i]j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

10 30 40 40

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40


40/70



for i = 2 to n {

key = A[i]j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

10 30 30 40

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 30


41/70



for i = 2 to n {

key = A[i]j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

10 30 30 40

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 30


42/70



for i = 2 to n {

key = A[i]j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

10 30 30 40

1 2 3 4

i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 30


43/70



for i = 2 to n {

key = A[i]j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

10 30 30 40

1 2 3 4

i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 30


44/70



for i = 2 to n {

key = A[i]j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

10 20 30 40

1 2 3 4

i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 20


45/70



for i = 2 to n {

key = A[i]j = i - 1;


A[j+1] = A[j]

j = j - 1

}A[j+1] = key

}

}

10 20 30 40

1 2 3 4

i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 20

Done!


46/70

Insertion Sort


for i = 2 to n {

key = A[i]

j = i - 1;

while (j > 0) and (A[j] > key) {A[j+1] = A[j]

j = j - 1

}

A[j+1] = key

}

}

How many times willthis loop execute?


47/70

T(n) for Insertion Sort

Worst case?

Best case?


48/70


49/70

Big O Fact

A polynomial of degree kis O(nk)

Proof:

Suppose f(n) = bknk + bk-1n

k-1 + + b1n + b0 Let ai = | bi |

f(n) aknk + ak-1n

k-1 + + a1n + a0

k

i

k

k

i

i

k

cnann

nan


50/70

Lower Bound Notation

In general a function

f(n) is (g(n)) if positive constants c and n0 such

that 0 cg(n) f(n) n n0

Proof:

Suppose run time is an + b

Assume a and b are positive (what ifb is negative?)

an an + b Example: n3 = (n2)


51/70

Asymptotic Tight Bound

A functionf(n) is (g(n)) if positive constants c1, c2,

and n0 such that

c1 g(n)

f(n)

c2 g(n)

n

n0 Theorem

f(n) is (g(n)) iff f(n) is both O(g(n)) and (g(n))

Proof

Engineering

Drop low-order items; ignore leading constants

Example: 3n3 + 90n2 -5n + 9999 = (n3)


52/70

Using limits for comparing orders of growth

)1(2

1nn

00

)(

)(lim c

ng

nt

n

T(n) has a smaller order of growth than g(n)

T(n) has the same order of growth than g(n)

T(n) has a larger order of growth than g(n)

Example: compare the orders of growth of and n^2


53/70

Practical Complexity

0

250

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

f(n) = n

f(n) = log(n)f(n) = n log(n)

f(n) = n^2

f(n) = n^3

f(n) = 2^n


54/70


55/70


0

1000

1 3 5 7 9 11 13 15 17 19

f(n) = n


f(n) = n^2

f(n) = n^3

f(n) = 2^n


56/70

David Luebke

56


0

1000

2000

3000

4000

5000

1 3 5 7 9 11 13 15 17 19

f(n) = n


f(n) = n^2

f(n) = n^3

f(n) = 2^n


57/70

Properties of Big-O

f(n) O(f(n))

f(n) O(g(n)) iffg(n) (f(n))

Iff(n) O(g (n)) and g(n) O(h(n)) , thenf(n)

O(h(n))Note similarity with a b

Iff1(n) O(g1(n)) andf2(n) O(g2(n)) , then

f1(n) +f2(n)

O(max{g1(n), g2(n)})


58/70

Basic asymptotic efficiency classes1 constant

log n logarithmic

n linear

n log n n-log-n

n2 quadratic

n3 cubic

2n exponential

n! factorial


59/70

Analysis for Algorithms

Non-recursive algorithms

Recursive algorithms

Often, we focus on worst case

T(n) for nonrecursive algorithms


60/70

T(n) for nonrecursive algorithms

General Plan for Analysis

Decide on parameter n indicating input size

Identify algorithms basic operation

Determine worst, average, and bestcases forinput of size n

Set up a sum for the number of times the basicoperation is executed

Simplify the sum using standard formulas andrules


61/70

Useful summation formulas and rulesliu1 = 1+1++1 = u - l+ 1

In particular, 1in1 = n - 1 + 1 = n (n)

1in i= 1+2++n = n(n+1)/2 n2/2 (n2)

1in i2 = 12+22++n2 = n(n+1)(2n+1)/6 n3/3 (n3)

0in ai = 1 + a ++ an = (an+1 - 1)/(a - 1) for any a 1

In particular, 0in 2i = 20 + 21 ++ 2n = 2n+1 - 1 (2n )

(ai bi) = ai bi cai = cai liuai = limai+ m+1iuai

Example 1: Maximum element


62/70

Example 1: Maximum element

Example 2: Element uniqueness problem


63/70

Example 2: Element uniqueness problem


64/70

Example 3: Matrix multiplication

Plan for Analysis of Recursive Algorithms


65/70

Plan for Analysis of Recursive Algorithms

Decide on a parameter indicating an inputs size.

Identify the algorithms basic operation.

Check whether the number of times the basic op. is executedmay vary on different inputs of the same size. (If it may, the

worst, average, and best cases must be investigatedseparately.)

Set up a recurrence relation with an appropriate initialcondition expressing the number of times the basic op. is

executed.

Solve the recurrence (or, at the very least, establish itssolutions order of growth) by backward substitutions or

another method.

Example 1: Recursive evaluation of n!


66/70

Example 1: Recursive evaluation ofn!

Definition: n ! = 1 2 (n-1) n for n 1 and 0! = 1

Recursive definition ofn!: F(n) = F(n-1) n for n 1 and

F(0) = 1

Size:

Basic operation:

Recurrence relation:

Example 2: The Tower of Hanoi Puzzle


67/70

Example 2: The Tower of Hanoi Puzzle

1

2

3

Recurrence for number of moves:

ndisks of different sizes and 3 pegs; the goal is to move all the disks to the thirdpeg using the 2nd one as an auxiliary if necessary. Move on disk at a time and do

not place a larger disk on top of a smaller one!


68/70

The Tower of Hanoi Puzzle

T(n) = 2T(n-1) + 1

T(1) = 1


69/70

Example 3: Counting #bits


70/70

Smoothness rule

In the preceding example, we assume n = 2k

and then take advantage of the theorem

called the smoothness rule:

Claims that: under very broad assumptions theorder of growth observed for n= 2kgives a correct

answer about the order of growth for all values of

n.

alg efficiency

Documents