analysis of algorithms - virginia...

Computational Tractability Asymptotic Order of Growth Common Running Times

Analysis of Algorithms

T. M. Murali

January 21, 2016

Force-add: Visit https://www.cs.vt.edu/S16Force-Adds before 3:15pm todayand use the password 4104tmm$

T. M. Murali January 21, 2016 Analysis of Algorithms

https://www.cs.vt.edu/S16Force-Adds


What is Algorithm Analysis?

I Measure resource requirements: how does the amount of time and space analgorithm uses scale with increasing input size?

I How do we put this notion on a concrete footing?

I What does it mean for one function to grow faster or slower than another?

I Goal: Develop algorithms that provably run quickly and use low amounts ofspace.



Worst-case Running Time

I We will measure worst-case running time of an algorithm.

I Bound the largest possible running time the algorithm over all inputs of sizen, as a function of n.

I Why worst-case? Why not average-case or on random inputs?

I Input size = number of elements in the input. Values in the input do notmatter, except for specic algorithms.

I Assume all elementary operations take unit time: assignment, arithmetic on axed-size number, comparisons, array lookup, following a pointer, etc.







I Input size = number of elements in the input.

Values in the input do notmatter, except for specic algorithms.








I Input size = number of elements in the input. Values in the input do notmatter, except for specic algorithms.




Polynomial Time

I Brute force algorithm: Check every possible solution.

I What is a brute force algorithm for sorting: given n numbers, permute themso that they appear in increasing order?

I Try all possible n! permutations of the numbers.I For each permutation, check if it is sorted.I Running time is nn!. Unacceptable in practice!

I Desirable scaling property: when the input size doubles, the algorithm shouldonly slow down by some constant factor c.

I An algorithm has a polynomial running time if there exist constants c > 0and d > 0 such that on every input of size n, the running time of thealgorithm is bounded by cnd steps.

Denition

An algorithm is ecient if it has a polynomial running time.



Polynomial Time



I Try all possible n! permutations of the numbers.I For each permutation, check if it is sorted.

I Running time is nn!. Unacceptable in practice!



Denition




Polynomial Time



I Try all possible n! permutations of the numbers.I For each permutation, check if it is sorted.I Running time is nn!. Unacceptable in practice!



Denition




Comparing Functions

I Assume all functions take only positive arguments and values.

I Dierent algorithms for the same problem may have dierent (worst-case)running times.

I Example of sorting:

bubble sort, insertion sort, quick sort, merge sort, etc.

I Bubble sort and insertion sort take roughly n2 comparisons while quick sortand merge sort take roughly n log

2n comparisons.

I Roughly hides potentially large constants, e.g., running time of merge sort

may in reality be 100n log2n.

I How can make statements such as the following?I 100n log

2n ≤ n2

I 10000n ≤ n2

I 5n2 − 4n ≥ 1000n log n



Comparing Functions

I Assume all functions take only positive arguments and values.

I Dierent algorithms for the same problem may have dierent (worst-case)running times.

I Example of sorting: bubble sort, insertion sort, quick sort, merge sort, etc.

I Bubble sort and insertion sort take roughly n2 comparisons while quick sortand merge sort take roughly n log

2n comparisons.

I Roughly hides potentially large constants, e.g., running time of merge sort

may in reality be 100n log2n.

I How can make statements such as the following?I 100n log

2n ≤ n2

I 10000n ≤ n2

I 5n2 − 4n ≥ 1000n log n



10000n ≤ n2

0 200 400 600 800 1,000

0

0.2

0.4

0.6

0.8

1

·107

n

10000n vs. n2

10000n

n2



10000n ≤ n2

0 0.5 1 1.5 2

·104

0

1

2

3

4

·108

n

10000n vs. O(n2)

10000n

n2



Upper Bound

Denition

Asymptotic upper bound: A function f (n) is O(g(n)) if

there exist constant

s

c > 0 and n0 ≥ 0 such that

for all n

≥ n0

, we have f (n) ≤

c

g(n).

0 0.5 1 1.5 2

·104

0

1

2

3

4

·108

n

10000n is O(n2),

c = 1, n0 = 10000

10000n

n2



Upper Bound

Denition

Asymptotic upper bound: A function f (n) is O(g(n)) if there exists constant

s

c > 0

and n0 ≥ 0

such that for all n

≥ n0

, we have f (n) ≤ cg(n).

0 0.5 1 1.5 2

·104

0

1

2

3

4

·108

n

10000n is O(n2),

c = 1, n0 = 10000

10000n

n2



Upper Bound

Denition

Asymptotic upper bound: A function f (n) is O(g(n)) if there exist constantsc > 0 and n0 ≥ 0 such that for all n ≥ n0, we have f (n) ≤ cg(n).

0 0.5 1 1.5 2

·104

0

1

2

3

4

·108

n

10000n is O(n2),

c = 1, n0 = 10000

10000n

n2



Upper Bound

Denition

Asymptotic upper bound: A function f (n) is O(g(n)) if there exist constantsc > 0 and n0 ≥ 0 such that for all n ≥ n0, we have f (n) ≤ cg(n).

0 0.5 1 1.5 2

·104

0

1

2

3

4

·108

n

10000n is O(n2), c = 1, n0 = 10000

10000n

n2



100n log2 n and n2

0 500 1,000 1,500 2,000

0

1

2

3

4

·106

n

100n log2n is O(n2),

c = 1, n0 = 1500

100n log2n

n2



100n log2 n and n2

0 500 1,000 1,500 2,000

0

1

2

3

4

·106

n

100n log2n is O(n2), c = 1, n0 = 1500

100n log2n

n2



100n log2 n and n2

0 20 40 60 80 100

0

0.2

0.4

0.6

0.8

1

·106

n

100n log2n is O(n2), c = 100, n0 = 1

100n log2n

100n2



Lower Bound

Denition

Asymptotic lower bound: A function f (n) is Ω(g(n)) if

there exist constant

s

c > 0 and n0 ≥ 0 such that

for all n

≥ n0

, we have f (n) ≥

c

g(n).

0 500 1,000 1,500 2,000

0

500

1,000

1,500

2,000

n

n log2n/10 is Ω(n), c = 1, n0 = 1024

n log n/10n



Lower Bound

Denition

Asymptotic lower bound: A function f (n) is Ω(g(n)) if there exists constant

s

c > 0

and n0 ≥ 0

such that for all n

≥ n0

, we have f (n) ≥ cg(n).

0 500 1,000 1,500 2,000

0

500

1,000

1,500

2,000

n

n log2n/10 is Ω(n), c = 1, n0 = 1024

n log n/10n



Lower Bound

Denition

Asymptotic lower bound: A function f (n) is Ω(g(n)) if there exist constantsc > 0 and n0 ≥ 0 such that for all n ≥ n0, we have f (n) ≥ cg(n).

0 500 1,000 1,500 2,000

0

500

1,000

1,500

2,000

n

n log2n/10 is Ω(n), c = 1, n0 = 1024

n log n/10n



Lower Bound

Denition


0 20 40 60 80 100

0

20

40

60

80

100

n

n log2n/10 and Ω(n)

n log n/10n

0 500 1,000 1,500 2,000

0

500

1,000

1,500

2,000

n

n log2n/10 is Ω(n), c = 1, n0 = 1024

n log n/10n



Lower Bound

Denition


0 500 1,000 1,500 2,000

0

500

1,000

1,500

2,000

n

n log2n/10 is Ω(n), c = 1, n0 = 1024

n log n/10n



Meaning of Lower Bound in Dierent Contexts

I Functions:

n is a lower bound for n log n/10, i.e., n log n/10 = Ω(n). Thisstatement is purely about these two mathematical functions withoutrelevance to any algorithm or problem.

I Algorithms: The lower bound on the running time of bubble sort is Ω(n2).There is some input of n numbers that will cause bubble sort to take at leastΩ(n2) time, e.g., input the numbers in decreasing order.

I Problems: The problem of sorting n numbers has a lower bound ofΩ(n log n). For any comparison-based sorting algorithm, there is at least oneinput for which that algorithm will take Ω(n log n) steps.




I Functions: n is a lower bound for n log n/10, i.e., n log n/10 = Ω(n).

Thisstatement is purely about these two mathematical functions withoutrelevance to any algorithm or problem.






I Functions: n is a lower bound for n log n/10, i.e., n log n/10 = Ω(n). Thisstatement is purely about these two mathematical functions withoutrelevance to any algorithm or problem.







I Algorithms: The lower bound on the running time of bubble sort is Ω(n2).

There is some input of n numbers that will cause bubble sort to take at leastΩ(n2) time, e.g., input the numbers in decreasing order.






I Algorithms: The lower bound on the running time of bubble sort is Ω(n2).There is some input of n numbers that will cause bubble sort to take at leastΩ(n2) time, e.g.,

input the numbers in decreasing order.







I Problems: The problem of sorting n numbers has a lower bound ofΩ(n log n).

For any comparison-based sorting algorithm, there is at least oneinput for which that algorithm will take Ω(n log n) steps.



Tight Bound

Denition

Asymptotic tight bound: A function f (n) is Θ(g(n)) if f (n) is O(g(n)) and f (n)is Ω(g(n)).

I In all these denitions, c and n0 are constants independent of n.

I Abuse of notation: say g(n) = O(f (n)), g(n) = Ω(f (n)), g(n) = Θ(f (n)).



Properties of Asymptotic Growth Rates

Transitivity I If f = O(g) and g = O(h), then f = O(h).I If f = Ω(g) and g = Ω(h), then f = Ω(h).I If f = Θ(g) and g = Θ(h), then f = Θ(h).

Additivity I If f = O(h) and g = O(h), then f + g = O(h).I Similar statements hold for lower and tight bounds.I If k is a constant and there are k functions

fi = O(h), 1 ≤ i ≤ k, then f1 + f2 + . . .+ fk = O(h).I If f = O(g), then f + g = Θ(g).





Additivity I If f = O(h) and g = O(h), then f + g = O(h).I Similar statements hold for lower and tight bounds.

I If k is a constant and there are k functionsfi = O(h), 1 ≤ i ≤ k, then f1 + f2 + . . .+ fk = O(h).

I If f = O(g), then f + g = Θ(g).






fi = O(h), 1 ≤ i ≤ k,

then f1 + f2 + . . .+ fk = O(h).I If f = O(g), then f + g = Θ(g).






fi = O(h), 1 ≤ i ≤ k, then f1 + f2 + . . .+ fk = O(h).

I If f = O(g), then f + g = Θ(g).






fi = O(h), 1 ≤ i ≤ k, then f1 + f2 + . . .+ fk = O(h).I If f = O(g), then f + g =

Θ(g).






fi = O(h), 1 ≤ i ≤ k, then f1 + f2 + . . .+ fk = O(h).I If f = O(g), then f + g = Θ(g).



Examples

I f (n) = pn2 + qn + r is

θ(n2). Can ignore lower order terms.

I Is f (n) = pn2 + qn + r = O(n3)?

I f (n) =∑

0≤i≤dain

i = O(nd), if d > 0 is an integer constant and ad > 0.

I O(nd ) is the denition of polynomial time.

I Is an algorithm with running time O(n1.59) a polynomial-time algorithm?

I O(loga n) = O(logb n) for any pair of constants a, b > 1.

I For every constant x > 0, log n = O(nx).

I For every constant r > 1 and every constant d > 0, nd = O(rn).



Examples

I f (n) = pn2 + qn + r is θ(n2). Can ignore lower order terms.

I Is f (n) = pn2 + qn + r = O(n3)?

I f (n) =∑

0≤i≤dain









Examples


I Is f (n) = pn2 + qn + r = O(n3)?

I f (n) =∑

0≤i≤dain

i =

O(nd), if d > 0 is an integer constant and ad > 0.








Examples


I Is f (n) = pn2 + qn + r = O(n3)?

I f (n) =∑

0≤i≤dain









0 2 4 6 8 10 12

0

500

1,000

1,500

2,000

n

Dierent functions of n

n

n log n

n2

n3

2n



0 5 10 15 20

0

5

10

15

20

n

More functions of n

n

log2n

log3n

n0.5



Linear Time

I Running time is at most a constant factor times the size of the input.

I Finding the minimum, merging two sorted lists.

I Computing the median (or kth smallest) element in an unsorted list.Median-of-median algorithm.

I Sub-linear time. Binary search in a sorted array of n numbers takes O(log n)time.



Linear Time



I Computing the median (or kth smallest) element in an unsorted list.

Median-of-median algorithm.




Linear Time




I Sub-linear time.

Binary search in a sorted array of n numbers takes O(log n)time.



Linear Time







O(n log n) Time

I Any algorithm where the costliest step is sorting.



Quadratic Time

I Enumerate all pairs of elements.

I Given a set of n points in the plane, nd the pair that are the closest.Surprising fact: will solve this problem in O(n log n) time later in thesemester.



Quadratic Time


I Given a set of n points in the plane, nd the pair that are the closest.

Surprising fact: will solve this problem in O(n log n) time later in thesemester.



Quadratic Time


I Given a set of n points in the plane, nd the pair that are the closest.Surprising fact: will solve this problem in O(n log n) time later in thesemester.



O(nk) Time

I Does a graph have an independent set of size k, where k is a constant, i.e.there are k nodes such that no two are joined by an edge?

I Algorithm: For each subset S of k nodes, check if S is an independent set. Ifthe answer is yes, report it.

I Running time is

O(k2(n

k

)) = O(nk).



O(nk) Time

I Does a graph have an independent set of size k, where k is a constant, i.e.there are k nodes such that no two are joined by an edge?

I Algorithm: For each subset S of k nodes, check if S is an independent set. Ifthe answer is yes, report it.

I Running time is O(k2(n

k

)) = O(nk).



Beyond Polynomial Time

I What is the largest size of an independent set in a graph with n nodes?

I Algorithm: For each 1 ≤ i ≤ n, check if the graph has an independent size ofsize i . Output largest independent set found.

I What is the running time? O(n22n).






I What is the running time?

O(n22n).






I What is the running time? O(n22n).


analysis of algorithms - virginia...

Documents