psu cs 311 – algorithm design and analysis dr. mohamed tounsi 1 cs 311 design and algorithms...
Post on 22-Dec-2015
216 views
TRANSCRIPT
1PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
CS 311 Design and Algorithms
Analysis
Dr. Mohamed Tounsi
2PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Problems and Algorithms
Abstractly, a problem is just a function
p : Problem instances -> Solutions
Example: Sorting a list of integersProblem instances = lists of integersSolutions = sorted lists of integersp : ` L-> Sorted version of L
An algorithm for p is a program which computes p.There are four related questions which warrant
consideration:
3PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Question 1: Given an algorithm A to solve a problem p, how “good” is A?
Question 2: Given a particular problem p for which several algorithms are known to exist, which is best?
Question 3: Given a problem p, how does one design good algorithms for p?
Problems and Algorithms
4PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Algorithm Analysis
The main focus of algorithm analysis in this course will be upon the “quality” of algorithms already known to be correct.
This analysis proceeds in two dimensions: Time complexity: The amount of time which
execution of the algorithm takes, usually specified as a function of the size of the input
Space complexity: The amount of space (memory) which execution of the algorithm takes, usually specified as a function of the size of the input.
Such analyses may be performed both experimentally and analytically.
5PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Designing Algorithms
It is important to study the process of designing good algorithms in the first place.
There are two principal approaches to algorithm design.
By problem: Study sorting algorithms, then scheduling algorithms, etc.
By strategy: Study algorithms by design strategy. Examples of design strategies include:
Divide-and-conquerThe greedy methodDynamic programmingBacktrackingBranch-and-bound
6PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Algorithm Definition
AlgorithmInput Output
An algorithm is a step-by-step procedure forsolving a problem in a finite amount of time.
7PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Running Time
Most algorithms transform input objects into output objects.
The running time of an algorithm typically grows with the input size.
Average case time is often difficult to determine.
We focus on the worst case running time.
Easier to analyze Crucial to applications such as
games, finance and robotics
0
20
40
60
80
100
120
Runnin
g T
ime
1000 2000 3000 4000
Input Size
best caseaverage caseworst case
8PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Experimental Studies
Write a program implementing the algorithm
Run the program with inputs of varying size and composition
Use a method like times() to get an accurate measure of the actual running time
Plot the results
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
0 50 100
Input Size
Tim
e (
ms)
9PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Limitations of Experiments
It is necessary to implement the algorithm, which may be difficult
Results may not be indicative of the running time on other inputs not included in the experiment.
In order to compare two algorithms, the same hardware and software environments must be used
10PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Theoretical Analysis
Uses a high-level description of the algorithm instead of an implementation
Characterizes running time as a function of the input size, n.
Takes into account all possible inputs Allows us to evaluate the speed of an
algorithm independent of the hardware/software environment
11PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Pseudocode
High-level description of an algorithm
More structured than English prose
Less detailed than a program
Preferred notation for describing algorithms
Hides program design issues
Algorithm arrayMax(A, n)Input array A of n integersOutput maximum element of A
currentMax A[0]for i 1 to n 1 do
if A[i] currentMax thencurrentMax A[i]
return currentMax
Example: find max element of an array
12PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Pseudocode Details
Control flow if … then … [else …] while … do … repeat … until … for … do … Indentation replaces braces
Method declarationAlgorithm method (arg [, arg…])
Input …
Output …
Method callvar.method (arg [, arg…])
Return valuereturn expression
Expressions Assignment
(like in Java) Equality testing
(like in Java)n2 Superscripts and other
mathematical formatting allowed
13PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
The Random Access Machine (RAM) Model
A CPU
An potentially unbounded bank of memory cells, each of which can hold an arbitrary number or character
01
2
Memory cells are numbered and accessing any cell in memory takes unit time.
14PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Primitive Operations
Basic computations performed by an algorithm
Identifiable in pseudocode Largely independent from the
programming language Exact definition not important (we
will see why later) Assumed to take a constant
amount of time in the RAM model
Examples: Evaluating an
expression Assigning a value to a
variable Indexing into an array Calling a method Returning from a
method
15PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Counting Primitive Operations
By inspecting the pseudocode, we can determine the maximum number of primitive operations executed by an algorithm, as a function of the input size
Algorithm arrayMax(A, n)
# operations
currentMax A[0] 2for i 1 to n 1 do 2 n
if A[i] currentMax then 2(n 1)currentMax A[i] 2(n 1)
{ increment counter i } 2(n 1)return currentMax 1
Total 7n 1
16PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Estimating Running Time
Algorithm arrayMax executes 7n 1 primitive operations in the worst case. Define:a = Time taken by the fastest primitive operationb = Time taken by the slowest primitive operation
Let T(n) be worst-case time of arrayMax. Thena (7n 1) T(n) b(7n 1)
Hence, the running time T(n) is bounded by two linear functions
17PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Growth Rate of Running Time
Changing the hardware/ software environment Affects T(n) by a constant factor, but Does not alter the growth rate of T(n)
The linear growth rate of the running time T(n) is an intrinsic property of algorithm arrayMax
18PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Growth Rates
Growth rates of functions:
Linear n Quadratic n2
Cubic n3
In a log-log chart, the slope of the line corresponds to the growth rate of the function
1E+01E+21E+41E+61E+8
1E+101E+121E+141E+161E+181E+201E+221E+241E+261E+281E+30
1E+0 1E+2 1E+4 1E+6 1E+8 1E+10n
T(n
)
Cubic
Quadratic
Linear
19PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Constant Factors
The growth rate is not affected by
constant factors or lower-order terms
Examples 102n 105 is a linear
function 105n2 108n is a
quadratic function
1E+01E+21E+41E+61E+8
1E+101E+121E+141E+161E+181E+201E+221E+241E+26
1E+0 1E+2 1E+4 1E+6 1E+8 1E+10n
T(n
)
Quadratic
Quadratic
Linear
Linear
20PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Big-Oh Notation
Given functions f(n) and g(n), we say that f(n) is O(g(n)) if there are positive constantsc and n0 such that
f(n) cg(n) for n n0
Example: 2n 10 is O(n) 2n 10 cn (c 2) n 10 n 10(c 2) Pick c 3 and n0 10
1
10
100
1,000
10,000
1 10 100 1,000n
3n
2n+10
n
21PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Big-Oh Example
Example: the function n2 is not O(n)
n2 cn n c The above inequality
cannot be satisfied since c must be a constant
1
10
100
1,000
10,000
100,000
1,000,000
1 10 100 1,000n
n 2̂
100n
10n
n
22PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
More Big-Oh Examples
7n-2
7n-2 is O(n)need c > 0 and n0 1 such that 7n-2 c•n for n n0
this is true for c = 7 and n0 = 1
3n3 + 20n2 + 53n3 + 20n2 + 5 is O(n3)need c > 0 and n0 1 such that 3n3 + 20n2 + 5 c•n3 for n
n0
this is true for c = 4 and n0 = 21 3 log n + log log n3 log n + log log n is O(log n)need c > 0 and n0 1 such that 3 log n + log log n c•log n
for n n0
this is true for c = 4 and n0 = 2
23PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Big-Oh and Growth Rate
The big-Oh notation gives an upper bound on the growth rate of a function
The statement “f(n) is O(g(n))” means that the growth rate of f(n) is no more than the growth rate of g(n)
We can use the big-Oh notation to rank functions according to their growth rate
f(n) is O(g(n)) g(n) is O(f(n))
g(n) grows more Yes No
f(n) grows more No Yes
Same growth Yes Yes
24PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Big-Oh Rules
If is f(n) a polynomial of degree d, then f(n) is O(nd), i.e.,1. Drop lower-order terms2. Drop constant factors
Use the smallest possible class of functions Say “2n is O(n)” instead of “2n is O(n2)”
Use the simplest expression of the class Say “3n 5 is O(n)” instead of “3n 5 is O(3n)”
25PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Asymptotic Algorithm Analysis
The asymptotic analysis of an algorithm determines the running time in big-Oh notation
To perform the asymptotic analysis We find the worst-case number of primitive operations executed
as a function of the input size We express this function with big-Oh notation
Example: We determine that algorithm arrayMax executes at most 7n 1
primitive operations We say that algorithm arrayMax “runs in O(n) time”
Since constant factors and lower-order terms are eventually dropped anyhow, we can disregard them when counting primitive operations
26PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Computing Prefix Averages
We further illustrate asymptotic analysis with two algorithms for prefix averages
The i-th prefix average of an array X is average of the first (i 1) elements of X:
A[i] X[0] X[1] … X[i])/(i+1)
Computing the array A of prefix averages of another array X has applications to financial analysis
0
5
10
15
20
25
30
35
1 2 3 4 5 6 7
X
A
27PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Prefix Averages (Quadratic) The following algorithm computes prefix averages in quadratic
time by applying the definition
Algorithm prefixAverages1(X, n)Input array X of n integersOutput array A of prefix averages of X #operations A new array of n integers nfor i 0 to n 1 do n
s X[0] nfor j 1 to i do 1 2 … (n 1)
s s X[j] 1 2 … (n 1)A[i] s (i 1) n
return A 1
28PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Arithmetic Progression
The running time of prefixAverages1 isO(1 2 …n)
The sum of the first n integers is n(n 1) 2
There is a simple visual proof of this fact
Thus, algorithm prefixAverages1 runs in O(n2) time
0
1
2
3
4
5
6
7
1 2 3 4 5 6
29PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Prefix Averages (Linear) The following algorithm computes prefix averages in linear
time by keeping a running sum
Algorithm prefixAverages2(X, n)Input array X of n integersOutput array A of prefix averages of X #operationsA new array of n integers ns 0 1for i 0 to n 1 do n
s s X[i] nA[i] s (i 1) n
return A 1
Algorithm prefixAverages2 runs in O(n) time
30PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
properties of logarithms:logb(xy) = logbx + logby
logb (x/y) = logbx - logby
logbxa = alogbx
logba = logxa/logxb properties of exponentials:
a(b+c) = aba c
abc = (ab)c
ab /ac = a(b-c)
b = a logab
bc = a c*logab
Summations Logarithms and Exponents
Proof techniques Basic probability
Math you need to Review *
31PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Relatives of Big-Oh
big-Omega f(n) is (g(n)) if there is a constant c > 0
and an integer constant n0 1 such that
f(n) c•g(n) for n n0
big-Theta f(n) is (g(n)) if there are constants c’ > 0 and c’’ > 0 and an integer
constant n0 1 such that c’•g(n) f(n) c’’•g(n) for n n0
little-oh f(n) is o(g(n)) if, for any constant c > 0, there is an integer constant n0 0
such that f(n) c•g(n) for n n0
little-omega f(n) is (g(n)) if, for any constant c > 0, there is an integer constant n0 0
such that f(n) c•g(n) for n n0
32PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Intuition for Asymptotic Notation
Big-Oh f(n) is O(g(n)) if f(n) is asymptotically less than or equal to g(n)
big-Omega f(n) is (g(n)) if f(n) is asymptotically greater than or equal to g(n)
big-Theta f(n) is (g(n)) if f(n) is asymptotically equal to g(n)
little-oh f(n) is o(g(n)) if f(n) is asymptotically strictly less than g(n)
little-omega f(n) is (g(n)) if is asymptotically strictly greater than g(n)
33PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Example Uses of the Relatives of Big-Oh
f(n) is (g(n)) if, for any constant c > 0, there is an integer constant n0 0 such that f(n) c•g(n) for n n0
need 5n02 c•n0 given c, the n0 that satifies this is n0 c/5 0
5n2 is (n)
f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0 1 such that f(n) c•g(n) for n n0
let c = 1 and n0 = 1
5n2 is (n)
f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0 1 such that f(n) c•g(n) for n n0
let c = 5 and n0 = 1
5n2 is (n2)
34PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Best Worst and Average Case
The worst case complexity of the algorithm is the function defined by the maximum number of steps taken on any instance of size n
The best case complexity of the algorithm is the function defined by the minimum number of steps taken on any instance of size n
Each of these complexities defines a numerical function time vs size
35PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Best Worst and Average Case (cont.)
36PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Exact Analysis is Hard !!
We have agreed that the best, worst and average case complexity of an algorithm is a numerical function of the size of the instances
However it is difficult to work with exactly because it is typically very complicated
Thus it is usually cleaner and easier to talk about upper and lower bounds of the function
This is where the big O notation comes in
37PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Formalization of the Concept of Order
The next task is to formalize the notion of one function being more difficult to compute than another, subject to the following assumptions:
The argument n defining the instance size is sufficiently large
Positive constant multipliers are ignored.
38PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Formalization of Concepts (cont)
Definition Let f : Complexity functions will never be negative
for usable arguments. In any case, behavior before no is not
significant for the asymptotic mathematical analysis.
39PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Big O Notation
Definition Let f :
It is said that “g is big-oh of f ”.
We write g = O( f ) or g in O( f )
The intuition is that g is “smaller” than f ; i.e., that g represents a lesser complexity.
40PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
, and
The value of n0 shown is the minimum possible value (any greater value would also work
41PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Sequential Searchvoid seqsearch(int n, const keytype S [ ], keytype x, index & location)
{location = 1;while (location <= n && S[location] != x) location ++; if (location > n) location=0;}
T(n) = N
42PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Binary Search
void binsearch (int n, const keytype S[], keytype x, index& location){ index low, high, mid;
low = 1; high = n; location = 0; while (low < = high && location = = 0){ mid = ∟(low + high)/2 ;⌋ if (x = = S[mid]) location = mid; else if (x < S[ mid ]) high = mid - 1; else low = mid + 1; }}
43PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Matrix Multiplication
void matrixmult (int n, const number A[][], const number B[][], number C[][])
{index i, j, k;
for (i=1; j<= n; j++) for (j=1; j<= n; j++) {
C[ i ] [ j ] = 0; for (k=1; k<= n; k++)
C[ i ] [ j ] = C[ i ] [ j ] + A [ i ] [ k ] * B [ k ] [ j ];
}}
44PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Exchange Sort
void exchangesort (int n, keytype S[])
{ index i, j; for (i=1;i<= 1; i++) for (j=i+l; j<= n; j++) if (S[j] < S[i]) exchange S[i] and S[j];}
45PSUCS 311 – Algorithm Design and Analysis Dr. Mohamed Tounsi
Add Array Members
number sum (int n, const number S[ ])
{ index i; number result; result = 0; for (i = 1; i <= n; i++) result = result + S[i]; return result;}