Algorithm Analysis

CS 201 Fundamental Structures of Computer Science

Introduction

• Once an algorithm is given for a problem and decided to be correct, an important step is to determine how much in the way of resources (such as time or space) the algorithm will require.

• In this chapter, we shall discuss– How to estimate the time required for a

program

Mathematical background

• The idea of the following definitions is to establish a relative order among functions.

• Given two functions, there are usually points where one function is smaller than the other function.– It does not make sense to claim f(N) < g(N)– Let us compare f(N) = 1000N and g(N) = N2

– Thus, we will compare their relative rates of growth

Mathematical background

• Definition 1:

0

0

when )()(such that

and constants positive are thereif ))(()(

nNNfcNT

ncNfONT

)()(

)()(

)(

)())(()(

NTNf

NfNT

Nf

NTNfONT

on boundupper an is Thus

thanfaster no rate aat grows

ofthat to equalor than less

is of rate growth thethat says

Mathematical background

• Definition 2:

)()(

)()(

)(

)())(()(

NTNg

NgNT

Ng

NTNgNT

on boundlower a is Thus

thanslower no rate aat grows

ofthat to equalor thangreater

is of rate growth thethat says

0

0

when )()(such that

and constants positive are thereif ))(()(

nNNgcNT

ncNgNT

Mathematical background

• Definition 3:

))(()( and ))(()(

ifonly and if ))(()(

NhNTNhONT

NhNT

ofthat to equals

of rate growth thethat says

)(

)())(()(

Nh

NTNhNT

Examples

• f(N)=N3 grows faster than g(N)=N2, sog(N) = O(f(N)) or f(N) = (g(N))

• f(N)=N2 and g(N)=2N2 grow at the same rate, sof(N) = O(g(N)) and f(N) = (g(N))

• If g(N)=2N2, g(N)=O(N4), g(N)=O(N3), g(N)=O(N2) are all technically correct, but the last one is the best answer.

• Do not say T(N)=O(2N2) or T(N)=O(N2+N). The correct form is T(N)=O(N2).

Mathematical background

• Rule 1:

• Rule 2:

• Rule 3:

))()(()()( (b)

))()(()()( (a)

then)),(()( and ))(()( If

21

21

21

NgNfONTNT

NgNfONTNT

NgONTNfONT

)()(

then , degree of polynomial a is )( IfkNNT

kNT

slowly. very grow logarithms that us tellsThis

.constant any for )(log kNONk

Examples

• Determine which of f(N)=N logN and g(N)=N1.5 grows faster.

Determine which of logN and N0.5 grows faster.

Determine which of log2N and N grows faster.

Since N grows faster than any power of a log, g(N) grows faster than f(N).

Examples

• Consider the problem of downloading a file over the Internet.– Setting up the connection: 3 seconds– Download speed: 1.5 Kbytes/second

• If a file is N kilobytes, the time to download is T(N) = N/1.5 + 3, i.e., T(N) = O(N).– 1500K file takes 1003 seconds– 750K file takes 503 seconds

• If the connection speed doubles, both times decrease, but downloading 1500K still takes approximately twice the time downloading 750K.

Typical growth rates

Function Name

c Constant

log N Logarithmic

log2 N Log-squared

N Linear

N log N

N2 Quadratic

N3 Cubic

2N Exponential

Plots of various algorithms

Algorithm analysis

• The most important resource to analyze is generally the running time– We assume that simple instructions (such as addition,

comparison, and assignment) take exactly one unit time• Unlike the case with real computers

– For example, I/O operations take more time compared to comparison and arithmetic operators

– Obviously, we do not have this assumption for fancy operations such as matrix inversion, list insertion, and sorting.

– We assume infinite memory– We do not include the time required to read the input

Algorithm analysis

• Typically, the size of the input is the main consideration– Worst-case performance represents a guarantee for

performance on any possible input– Average-case performance often reflects typical behavior– Best-case performance is often of little interest

• Generally, it is focused on the worst-case analysis– It provides a bound for all inputs– Average-case bounds are much more difficult to compute

Algorithm analysis

• Although using Big-Theta would be more precise, Big-Oh answers are typically given.

• Big-Oh answer is not affected by the programming language– If a program is running much more slowly than the

algorithm analysis suggests, there may be an implementation inefficiency

• e.g., This can occur in C++ when arrays are inadvertently copied in their entirety instead of passed with references.

Algorithm analysis

• If two programs are expected to take similar times, probably the best way to decide which is faster to code them both up and run them!

• On the other hand, we would like to eliminate the bad algorithmic ideas early by algorithm analysis– Although different instructions can take different

amounts of time (these would correspond to constants in Big-Oh notation), we ignore this difference in our analysis and try to find the upper bound of the algorithm

A simple example

• Estimate the running time of the following function

int sum(int n){

int partialSum = 0;

for (int i = 0; i <= n; i++)

partialSum += i * i * i;

return partialSum;

}

General rules

• Rule 1 – for loops: The running time of a for loop is at most the running time of the statements inside the for loop (including tests) times the number of iterations

• Rule 2 – nested loops: Analyze these inside out. The total running time of a statement inside a group of nested loops is the running time of the statement multiplied by the product of the sizes of all the loops

for (i = 0; i < n; i++) for (j = i; j < n; j++) k++;

General rules

• Rule 3 – consecutive statements: just add. for (i = 0; i < n; i++) a[i] = 0; for (i = 0; i < n; i++) for (j = 0; j < n; j++) a[i] += a[j] + i + j;

• Rule 4 – if/else: For the following, the running time is never more than that of the test plus the larger of that of S1 and S2 if (condition) S1 else S2

General rules

• If there are function calls, these must be analyzed first

• If there are recursive functions, be careful about their analyses. For some recursions, the analysis is trivial long factorial(int n){ if (n <= 1) return 1; return n * factorial(n-1); }

General rules• Let us analyze the following recursion

long fib(int n){

if (n <= 1)

return 1;

return fib(n-1) + fib(n-2);

}• • By induction, it is possible to show that

• So the running time grows exponentially. By using a for loop, the running time can be reduced substantially

2)2()1()( NTNTNT

NN NfibNfib )2/3()( and )3/5()(

Sequential search

index = -1;for (i = 0; i < N; i++) if (A[i] == value){ index = i; break; }

• Worst-case: It is the last element or it is not found– N iterations O(N)

• Best-case: It is the first element– 1 iteration O(1)

• Average case: We may have N different cases– (N + 3) / 2 iterations O(N)

Searching a value in a sorted array using binary search

int binarySearch(int *A, int N, int value){ int low = 0, high = N; while (low <= high){ int mid = (low + high) / 2; if (a[mid] < value) low = mid + 1; else if (a[mid] > value) high = mid - 1; else return mid; } return -1;}

Iterative solution of raising an integer to a power

long pow1(long x, long n){

long result = 1;

for (int i = 1; i <= n; i++)

result = result * x;

return result;

}

Recursive solution of raising an integer to a power

long pow2(long x, long n){ if (n == 0) return 1; if (n == 1) return x; if (n % 2 == 0) return pow2(x*x,n/2); else return pow2(x*x,n/2) * x;}