algorithm analysis1 5simeon2m/cs210/lecture_notes...algorithm analysis what is an algorithm? an...
TRANSCRIPT
Algorithm Analysis
What is an algorithm?
An algorithm is a step-by-step procedure for accomplishing some
end.
Why do we want to analyze it?
To be able to draw conclusions about how the implementation of
that algorithm will perform in general.
Problem
Cannot make determination from a run on a single machine
What can we analyze?
• The running time of a program as a function of its
inputs
• The total or maximum memory space needed for program data
• The total size of the program code
• Whether the program correctly computes the desired result
• The complexity of the program - e.g how easy is it to read,
understand and modify
1
• The robustness of the program - e.g how well does it deal with
unexpected or erroneous inputs?
Performance of a Computer
Determined by
• the hardware - processor, memory, disk available
• programming language used
• compiler/interpreter used
• operating system
Problem
• difficult
• time-consuming
• technology changes
Detailed Model of the Computer
• Independent of the underlying hardware and software
• General model
Basic Axioms
Axiom 1
The time required to fetch an integer operand from memory is a
constant, Tfetch, and the time required to store an integer result
in memory is a constant, Tstore.
2
y=x;
running time: Tfetch + Tstore
Axiom 2
The times required to perform elementary operations on integers,
such as addition, subtraction, multiplication, division, and com-
parison, are all constants. These are denoted as T+, T−, T×, T÷,
and T<.
y=y+1;
running time = 2Tfetch + T+ + Tstore
Axiom 3
The time required to call a function is a constant, Tcall, and the
time to return from a function is a constant, Treturn.
Axiom 4
The time required to pass an integer argument to a function is the
same as the time required to store an integer in memory, Tstore.
y=f(x)
running time = Tfetch + 2Tstore + Tcall + Tf(x)
where Tf(x) is the running time of function f(x) for input x.
Example
Analyze the running time of a program to compute the following
arithmetic summationn
∑
i=1
i
3
1 i n t Sum( in t n)2 3 i n t r e s u l t = 0 ;4 f o r ( i n t i =0; i<=n ; ++i )5 r e s u l t += i ;6 return r e s u l t ;7
Table 1: Computing Running TimeStatement Time Code
3 Tfetch + Tstore result = 0
4a Tfetch + Tstore i = 1
4b (2Tfetch + T<) × (n + 1) i ≤ n
4c (2Tfetch + T+ + Tstore) × n ++i
5 (2Tfetch + T+ + Tstore) × n result += i
6 Tfetch + Treturn return result
TOTAL = (6Tfetch + 2Tstore + T< + 2T+) × n
= +(5Tfetch + 2Tstore + T< + Treturn)
T (n) = t1 + t2n
Axiom 5
The time required for the address calculation implied by an array
subscripting operation, e.g. a[i], is a constant, T[.]. This time does
not include the time to compute the subscript expression, nor does
it include the time to access (i.e. fetch or store) the array element.
y = a[i]
running time = 3Tfetch + Tstore
Example
Finding the largest element in an array
max0≤i<n
ai
4
1 i n t FindMaximum( in t a [ ] , i n t n )2 3 i n t r e s u l t = a [ 0 ] ;4 f o r ( i n t i = 1 ; i < n ; ++i )5 i f ( a [ i ] > r e s u l t )6 r e s u l t = a [ i ] ;7 return r e s u l t ;8
Table 2: Computing Running TimeStatement Time
3 3Tfetch + T[.] + Tstore
4a Tfetch + Tstore
4b (2Tfetch + T<) × n4c (2Tfetch + T+ + Tstore) × (n − 1)5 (4Tfetch + T[.] + T<) × (n − 1)6 3Tfetch + T[.] + Tstore×?7 Tfetch + Tstore
Line 6 executed only if
ai > (max0≤j<iaj)
? - depends on the actual elements of the array, a0, a1, ..., an−1
T (n, a0, a1, ..., an−1) = t1 + t2n +∑n−1
i=1 t3
where
t1 = 2Tstore − Tfetch − T+ − T<
t2 = 8Tfetch + 2T< + T[.] + T+ + Tstore
t3 = 3Tfetch + T[.] + Tstore
Average Running Times
If a program is run a large number of times on a selection of ran-
dom inputs of length n, what will the average running time be?
5
Taverage(n) = t1 + t2n +∑n−1
i=1 pit3
pi is the probability that line 6 is executed.
If the array is ordered smallest to largest, probability is 1.
If the array is ordered largest to smallest, probability is 0.
If the order is unknown, line 6 is only executed if ai is the largest
of the i + 1 values a0, a1, ..., ai
pi = 1i+1
Taverage(n) = t1 + t2n +n−1∑
i=1
pit3
= t1 + t2n + t3
n−1∑
i=1
1
i + 1
= t1 + t2n + t3(
n∑
i=1
1
i− 1)
= t1 + t2n + t3(Hn − 1)
where Hn =∑n
i=11i is the nth harmonic number
Hn − Hn−1 = 1n
Hn−1 is approximately ln n for n > 1
Error generated by the approximation is called Euler’s constant,
γ.
Hn ≈ ln n + γ
6
Taverage(n) = t1 + t2n + t3(Hn − 1)
≈ t1 + t2n + t3(ln n + γ − 1)
≈ (t1 + t3(γ − 1)) + t2n + t3(ln n)
Best and Worst-Case Running Time
Tworst(n) = t1 + t2n +n−1∑
i=1
pit3
= t1 + t2n + t3
n−1∑
i=1
1
= t1 + t2n + t3(n − 1)
= (t1 − t3) + (t2 + t3) × n
Tbest(n) = t1 + t2n +
n−1∑
i=1
pit3
= t1 + t2n + t3 × 0
= t1 + t2n
Simplified Model
Let T be the clock cycle of a machine. e.g Tfetch = kT
Assume
• All timing parameters expressed in units of clock cycles. Thus
T = 1
7
• k is assumed to be the same for all parameters, thus k = 1
8
Example
Compute the running time of a geometric series summation
n∑
i=0
xi.
1 i n t GeometricSeriesSum ( in t x , i n t n )2 3 i n t sum = 0 ;4 f o r ( i n t i = 0 ; i <= n ; ++i )5 6 i n t prod = 1 ;7 f o r ( i n t j = 0 ; j< i ; ++j )8 prod ∗= x ;9 sum += prod ;
10 11 return sum ;12
Table 3: Computing Running TimeStatement Time
3 24a 24b 3(n + 2)4c 4(n + 1)6 2(n + 1)7a 2(n + 1)7b 3
∑n
i=0(i + 1)7c 4
∑n
i=0 i8 4
∑n
i=0 i9 4(n + 1)11 2
Total 112n2 + 47
2n + 27
Sn =
n∑
i=1
i =n(n + 1)
2
1 i n t GeometricSeriesSum ( in t x , i n t n )
9
2 3 i n t sum = 0 ;4 f o r ( i n t i = 0 ; i <= n ; ++i )5 sum = sum ∗ x + 1 ;6 return sum ;7
Table 4: Computing Running TimeStatement Time
3 24a 24b 3(n + 2)4c 4(n + 1)5 6(n + 1)6 2
Total 13n + 22
10
1 i n t GeometricSeriesSum ( in t x , i n t n )2 3 return (Power (x , n+1) − 1) /(x − 1)4
1 i n t Power ( i n t x , i n t n )2 3 i f (n == 0)4 return 1 ;5 e l s e i f (n % 2 == 0)6 return Power ( x∗x , n /2 ) ;7 e l s e8 return x∗ Power (x∗x , n /2 ) ;9
Running Time: 19(⌊log2(n + 1)⌋ + 1) + 18
0
200
400
600
800
1000
0 20 40 60 80 100
T(n
)
n
13*x + 225.5*x*x + 23.5*x + 27
19*(floor(log(x+1)+1)) + 18
Asymptotic Notation
What does it mean to say one function, TA(n) is better thananother, TB(n)?
If the size of the problem is known. i.e. n0, then if TA(n)
If the problem size is not known, if it can be shown that ∀n ≥ 0 :
11
TA(n) ≤ TB(n) then A is better than B
Consider the asymptotic behavior of the two functions instead.
Asymptotic Upper Bound - Big Oh
Definition
Consider a function f(n) which is non-negative for all integers n ≥
0. We say that “f(n) is big oh g(n),” which we write f(n) =
O(g(n)), if there exists an integer n0 and a constant c > 0 such
that for all integers n ≥ n0, f(n) ≤ cg(n).
Example
Consider the function f(n) = 8n + 128, show that f(n) = O(n2).
We need to find an integer n0 and a constant c > 0 such that for
all integers n ≥ n0, f(n) ≤ cn2.
Let c = 1 then
f(n) ≤ cn2 → 8n2 + 128 ≤ n2
→ 0 ≤ n2 − 8n − 128
→ 0 ≤ (n − 16)(n + 8)
Since (n + 8) > 0 for all values n ≥ 0, we can conclude that
(n0 − 16) ≥ 0, i.e. n0 = 16.
Thus we have c = 1, n0 = 16, f(n) ≤ cn2 for all integers n ≥ n0,
hence f9n) = O(n2).
Solutions for c = 1, 2, 4 and n0 = 16, 10.2, 6.7
12
0
100
200
300
400
500
0 5 10 15 20 25
y=f(
n)
n
4*x*x2*x*x
x*x8*x + 128
Big Oh Fallacies
1. Given that f1(n) = O(g(n)) and f2(n) = O(g(n)) then
f1(n) = f2(n).
e.g f1(n) = n and f1(n) = n2, both are O(n2), but not equal
2. If f(n) = O(g(n)), then g(n) = O−1(f(n)).
O−1 has no meaning since Big Oh is not a mathematical func-
tion
Properties of Big Oh
Theorem 1
If f1(n) = O(g1(n)) and f2(n) = O(g2(n)) then f1(n) + f2(n) =
O(max(g1(n), g2(n)).
13
Example
Let g1(n) = 1 and g2(n) = 2cos2(nπ/2) then
h(n) = max(g1(n), g2(n))
= max(1, 2cos2(nπ/2)
h(n) =
1 n is even
2 n is odd
Theorem 2
If f(n) = f1(n) + f2(n) in which f1(n) and f2(n) are both non-
negative for all integers n ≥ 0 such that limn→∞f2(n)/f1(n) = L
for some limit L ≥ 0, then f(n) = O(f1(n)).
Example
Consider f1(n) = n3 and f2(n) = n2
limn→∞f2(n)
f1(n)= limn→∞
n2
n3
= limn→∞1
n= 0
Thus f1(n) + f2(n) = n3 + n2 = O(n3).
Conclusion: The sum of a series of powers of n is
O(nm) where m is the largest power of n in the sum-
mation.
14
Theorem 3
If f1(n) = O(g1(n)) and f2(n) = O(g2(n)), then
f1(n) × f2(n) = O(g1(n) × g2(n))
Example
Consider f1(n) = n3 +n2 +n+1 and f2(n) = n2 +n+1, then by
Theorem 2 we know that f1(n) = O(n3) and f2(n) = O(n2) thus
f1(n) × f2(n) = O(n3 × n2) = O(n5)
Theorem 4
If f1(n) = O(g1(n)) and g2(n) is a function whose value is non-
negative for integers n ≥ 0, then
f1(n) × g2(n) = O(g1(n) × g2(n))
Theorem 5
If f(n) = O(g(n)) and g(n) = O(h(n)) then f(n) = O(h(n)).
Theorem 6
Consider a polynomial in n of the form
f(n) =m
∑
i=0
aini
= amnm + am−1nm−1 + ... + a2n
2 + a1n + a − 0
where am > 0. Then f(n) = O(nm).
15
Theorem 7
For every integer k ≥ 1, logkn = O(n). log n diverges as n gets
large. log n < n for all integers n ≥ 0. Hence log n = O(n).
0
5
10
15
20
25
0 5 10 15 20 25
y=f(
n)
n
log(x)x
Tight Big Oh Bounds
Consider a function f(n) = O(g(n)). If for every function h(n)
such that f(n) = O(h(n)) it is also true that g(n) = O(h(n)),
then we say that g(n) is a tight asymptotic bound on f(n).
Example
Consider f(n) = 8n + 128, we showed earlier f(n) = O(n2) but
we know from earlier theorems that f(n) = O(n). Thus O(n) is a
tighter bound on the asymptotic behavior of f(n) than is O(n2).
16
Expression NameO(1) constantO(logn) logarithmicO(log2n) log squaredO(n) linearO(nlogn) nlognO(n2) quadraticO(n3) cubicO(2n) exponential
Conventions for Writing Big Oh Expressions
• Drop all but the most significant terms. Thus instead of
O(n2 + nlogn + n) we write O(n2).
• Drop constant coefficients. Instead of O(3n2) we write O(n2).
• Use a tight asymptotic bound. Instead of f(n) = n = O(n3),
write f(n) = O(n).
0
5
10
15
20
25
0 5 10 15 20 25
y=f(
n)
n
1log(x)
log(x)*log(x)x
x*log(x)x*x
x*x*x2**x
17
Asymptotic Lower Bound - Omega
Consider a function f(n) which is non-negative for all integers
n ≥ 0. We say that “f(n) is omega g(n)” which we write f(n) =
Ω(g(n)), if there exists an integer n0 and a constant c > 0 such
that for all integers n ≥ n0, f(n) ≥ cg(n).
Example
Consider the function f(x) = 5n2−64n+256, f(n) is non-negative
for all integers n ≥ 0. Show that f(n) = Ω(n2). Need to find an
integer n0 and a constant c > 0 such that for all integers n ≥ n0,
f(n) ≥ cn2.
Suppose c = 1,
f(n) ≥ cn2 → 5n2 − 64n + 256 ≥ n2
→ 4n2 − 64n + 256 ≥ 0
→ 4(n − 8)2 ≥ 0
Since (n − 8)2 > 0 for all values n ≥ 0, we conclude n0 = 0.
Theorem 8
Consider a polynomial in n of the form
f(n) =m
∑
i=0
aini
= amnm + am−1nm−1 + ... + a2n
2 + a1n + a0
where am > 0. Then f(n) = Ω(nm).
18
0
200
400
600
800
1000
1200
1400
0 5 10 15 20 25
y=f(
n)
n
5*x*x-64*x+2562*x*x
x*x
Theta
Consider a function f(n) which is non-negative for all integers
n ≥ 0. We say that “f(n) is theta g(n)” which we write f(n) =
Θ(g(n)), if and only if f(n) is O(g(n)) and f(n) is Ω(g(n)).
Example
Previously showed that for a polynomial, f(n) = amnm+am−1nm−1+
... + a2n2 + a1n + a0, is both O(nm) and Ω(nm). Thus we can say
f(n) = Θ(nm).
Little Oh
Consider a function f(n) which is non-negative for all integers
n ≥ 0. We say that “f(n) is little oh g(n)” which we write
f(n) = o(g(n)), if and only if f(n) is O(g(n)) but f(n) is not
Θ(g(n)).
19
Example
Consider the function f(n) = n + 1. Clearly f(n) = O(n2). Also
f(n) 6= Ω(n2) since no matter what c is chosen, for large enough
n, cn2 ≥ n + 1. Thus f(n) = n + 1 = o(n2).
1 i n t Horner ( i n t a [ ] , i n t n , i n t x )2 3 i n t r e s u l t = a [ n ] ;4 f o r ( i n t i = n−1; i >= 0 ; −− i )5 r e s u l t = r e s u l t ∗ x + a [ i ] ;6 return r e s u l t ;7
Statement Detailed Model Simple Model Big Oh3 3Tfetch + T[.] + Tstore 5 O(1)4a 2Tfetch + T− + Tstore 4 O(1)4b (2Tfetch + T<) × (n + 1) 3n+3 O(n)4c (2Tfetch + T− + Tstore) × n 4n O(n)5 (5Tfetch + T[.] + T+ + T× + Tstore) × n 9n O(n)6 Tfetch + Treturn 2 O(1)Total Ttotal 16n + 14 O(n)
Ttotal = (9Tfetch + T[.] + T+ + T< + T− + T× + 2Tstore) × n +
= 8Tfetch + T[.] + T< + T− + Treturn + 2Tstore
20
Rules for Big Oh Analysis of Running Time
Rule 1: Sequential Composition
The worst case running time of a sequence of C++ statements
such as
S1;
S2;
.
.
.
Sm;
is O(max(T1(n), T2(n), ..., Tm(n))), where the running time of
Si, the ith statement in the sequence, is O(Ti(n)).
Rule 2: Iteration
The worst case running time of a C++ for loop such as
for (S1; S2; S3)
S4;
is O(max(T1(n), T2(n)×(I(n)+1), T3(n)×I(n), T4(n)×I(n))),
where the running time of a statement Si is O(Ti(n)), for i =
1, 2, 3, 4 and I(n) is the number of iterations executed in the worst
case.
Example
1 f o r ( i n t i = 0 ; i <n ; i++)2 S 4 ;
21
S1 is int i = 0, so the running time is constant T1(n) = 1.
S2 is i < n, so the running time is constant T2(n) = 1.
S3 is + + 1, so its running time is a constant T3(n) = 1.
The number of iterations is I(n) = n.
According to Rule 2, the running time of this is O(max(1, 1× (n+
1), 1 × n, T4(n) × n)), which simplifies to O(max(n, T4(n) × n)).
If the loop does anything at all, its running time must be T4(n) =
Ω(1).
Loop body dominates and the running time is O(n × T4(n).
If number of iterations is unknown, the rule also applies if an upper
bound on the number of iterations is known, i.e I(n) = O(f(n)).
Running time would be O(max(T1(n), T2(n)× (f(n)+1), T3(n)×
f(n), T4(n) × f(n))).
Rule 3:Conditional Execution
The worst case running time of a C++ if-then-else such as
if(S1)
S2;
else
S3;
is O(max(T1(n), T2(n), T3(n))) the running time of statement
Si is O(Ti(n)), for i = 1, 2, 3.
22
Reality Check
Table 5: Actual Lower Bounds assuming a 100Mhz Clock, c = 1 Cycle and n0 = 0n = 1 n = 8 n = 1K n = 1024K
Ω(1) 10ns 10ns 10ns 10nsΩ(logn) 10ns 30ns 100ns 200nsΩ(n) 10ns 80ns 10.2µs 10msΩ(nlogn) 10ns 240ns 10.2µs 210msΩ(n2) 10ns 640ns 102µs 3.05hrsΩ(n3) 10ns 5.12µs 10.7s 365yrs
Ω(2n) 10ns 2.56µs 10293yrs 10105yrs
23