introduction to algorithms: asymptotic...
TRANSCRIPT
Asymptotic Notation
0that 0 f(n) cg(n) for all n n .
O-notation (upper bounds):
CS 421 - Analysis of Algorithms 3
Asymptotic Notation
0that 0 f(n) cg(n) for all n n .
O-notation (upper bounds):
CS 421 - Analysis of Algorithms 454 Fundamentals of the Analysis of Algorithm Efficiency
doesn'tmatter
nn0
cg(n)
t (n)
FIGURE 2.1 Big-oh notation: t (n) ∈ O(g(n)).
doesn'tmatter
nn0
cg(n)
t (n)
FIGURE 2.2 Big-omega notation: t (n) ∈ !(g(n)).
!-notationDEFINITION A function t (n) is said to be in !(g(n)), denoted t (n) ∈ !(g(n)), ift (n) is bounded below by some positive constant multiple of g(n) for all large n,
i.e., if there exist some positive constant c and some nonnegative integer n0 suchthat
t (n) ≥ cg(n) for all n ≥ n0.
The definition is illustrated in Figure 2.2.
Here is an example of the formal proof that n3 ∈ !(n2):
n3 ≥ n2 for all n ≥ 0,
i.e., we can select c = 1 and n0 = 0.
54 Fundamentals of the Analysis of Algorithm Efficiency
doesn'tmatter
nn0
cg(n)
t (n)
FIGURE 2.1 Big-oh notation: t (n) ∈ O(g(n)).
doesn'tmatter
nn0
cg(n)
t (n)
FIGURE 2.2 Big-omega notation: t (n) ∈ !(g(n)).
!-notationDEFINITION A function t (n) is said to be in !(g(n)), denoted t (n) ∈ !(g(n)), ift (n) is bounded below by some positive constant multiple of g(n) for all large n,
i.e., if there exist some positive constant c and some nonnegative integer n0 suchthat
t (n) ≥ cg(n) for all n ≥ n0.
The definition is illustrated in Figure 2.2.
Here is an example of the formal proof that n3 ∈ !(n2):
n3 ≥ n2 for all n ≥ 0,
i.e., we can select c = 1 and n0 = 0.
Asymptotic Notation
0that 0 f(n) cg(n) for all n n .
O-notation (upper bounds):
CS 421 - Analysis of Algorithms 5
where c = 1, n0 = 2
Asymptotic Notation
0that 0 f(n) cg(n) for all n n .
O-notation (upper bounds):
CS 421 - Analysis of Algorithms 6
where c = 1, n0 = 2functions, not values
funny, one-way equality
Asymptotic Notation
0that 0 f(n) cg(n) for all n n .
O-notation (upper bounds):
2.2 Asymptotic Notations and Basic Efficiency Classes 53
Indeed, the first two functions are linear and hence have a lower order of growththan g(n) = n2, while the last one is quadratic and hence has the same order ofgrowth as n2. On the other hand,
n3 ̸∈ O(n2), 0.00001n3 ̸∈ O(n2), n4 + n + 1 ̸∈ O(n2).
Indeed, the functions n3 and 0.00001n3 are both cubic and hence have a higherorder of growth than n2, and so has the fourth-degree polynomial n4 + n + 1.
The second notation, !(g(n)), stands for the set of all functions with a higheror same order of growth as g(n) (to within a constant multiple, as n goes to infinity).For example,
n3 ∈ !(n2),12n(n − 1) ∈ !(n2), but 100n + 5 ̸∈ !(n2).
Finally, "(g(n)) is the set of all functions that have the same order of growthas g(n) (to within a constant multiple, as n goes to infinity). Thus, every quadraticfunction an2 + bn + c with a > 0 is in "(n2), but so are, among infinitely manyothers, n2 + sin n and n2 + log n. (Can you explain why?)
Hopefully, this informal introduction has made you comfortable with the ideabehind the three asymptotic notations. So now come the formal definitions.
O-notationDEFINITION A function t (n) is said to be in O(g(n)), denoted t (n) ∈ O(g(n)),
if t (n) is bounded above by some constant multiple of g(n) for all large n, i.e., ifthere exist some positive constant c and some nonnegative integer n0 such that
t (n) ≤ cg(n) for all n ≥ n0.
The definition is illustrated in Figure 2.1 where, for the sake of visual clarity, n isextended to be a real number.
As an example, let us formally prove one of the assertions made in theintroduction: 100n + 5 ∈ O(n2). Indeed,
100n + 5 ≤ 100n + n (for all n ≥ 5) = 101n ≤ 101n2.
Thus, as values of the constants c and n0 required by the definition, we can take101 and 5, respectively.
Note that the definition gives us a lot of freedom in choosing specific valuesfor constants c and n0. For example, we could also reason that
100n + 5 ≤ 100n + 5n (for all n ≥ 1) = 105n
to complete the proof with c = 105 and n0 = 1.
Set Definition of Ο-notation
CS 421 - Analysis of Algorithms 9
Ο(𝑔 𝑛 ) is the set of all functions with a SMALLER or the SAME order of growth as 𝑔 𝑛 .
Set Definition of Ο-notation
CS 421 - Analysis of Algorithms 10
Ο(𝑔 𝑛 ) is the set of all functions with a SMALLER or the SAME order of growth as 𝑓(𝑛).
EXAMPLE: Ο(𝑛') = 5𝑛', 10𝑛- + 500, log𝑛, …
5𝑛' ∈ Ο(𝑛')Or put another way:
Asymptotic Notation
0that 0 f(n) cg(n) for all n n .
Ω-notation (lower bounds):
CS 421 - Analysis of Algorithms 12
Asymptotic Notation
0that 0 f(n) cg(n) for all n n .
Ω-notation (lower bounds):
CS 421 - Analysis of Algorithms 13
Asymptotic Notation
0that 0 f(n) cg(n) for all n n .
Ω-notation (lower bounds):
CS 421 - Analysis of Algorithms 14
where c = 1, n0 = 16
Asymptotic Notation
0that 0 f(n) cg(n) for all n n .
Ω-notation (lower bounds):
CS 421 - Analysis of Algorithms 15
Set Definition of Ω-notation
CS 421 - Analysis of Algorithms 16
Ω(𝑔 𝑛 ) is the set of all functions with a LARGER or the SAME order of growth as 𝑔 𝑛 .
EXAMPLE: Ω(𝑛-) = 5𝑛-, 10𝑛' + 500, 𝑛- ∗ log 𝑛, …
5𝑛- ∈ Ω(𝑛-)Or put another way:
CS 421 - Analysis of Algorithms 17
Θ-notation (tight bounds)Combine definitions of Ο and Ω:
𝑊𝑒𝑤𝑟𝑖𝑡𝑒:𝑓 𝑛 = Θ 𝑔 𝑛
𝑖𝑓𝑡ℎ𝑒𝑟𝑒𝑒𝑥𝑖𝑠𝑡𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡𝑠𝑐1 > 0, 𝑐2 > 0𝑎𝑛𝑑𝑛0 > 0
𝑠𝑢𝑐ℎ𝑡ℎ𝑎𝑡c1𝑔 𝑛 <= 𝑓 𝑛 <= 𝑐2𝑔(𝑛)
CS 421 - Analysis of Algorithms 18
Θ-notation (tight bounds)Combine definitions of Ο and Ω:
𝑊𝑒𝑤𝑟𝑖𝑡𝑒:𝑓 𝑛 = Θ 𝑔 𝑛
𝑖𝑓𝑡ℎ𝑒𝑟𝑒𝑒𝑥𝑖𝑠𝑡𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡𝑠𝑐1 > 0, 𝑐2 > 0𝑎𝑛𝑑𝑛0 > 0
𝑠𝑢𝑐ℎ𝑡ℎ𝑎𝑡c1𝑔 𝑛 <= 𝑓 𝑛 <= 𝑐2𝑔(𝑛)
CS 421 - Analysis of Algorithms 25
ο-notationO-notation like ≤.o-notation like <.𝑊𝑒𝑤𝑟𝑖𝑡𝑒: 𝑓 𝑛 = 𝜊 𝑔 𝑛
𝑓𝑜𝑟𝑎𝑛𝑦𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡𝑠𝑐 > 0,
𝑡ℎ𝑒𝑟𝑒𝑖𝑠𝑎𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡𝑛M > 0𝑠𝑢𝑐ℎ𝑡ℎ𝑎𝑡
0 < 𝑓(𝑛) < 𝑐𝑔(𝑛)
CS 421 - Analysis of Algorithms 26
ο-notationO-notation like ≤.o-notation like <.𝑊𝑒𝑤𝑟𝑖𝑡𝑒: 𝑓 𝑛 = 𝜊 𝑔 𝑛
𝑓𝑜𝑟𝑎𝑛𝑦𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡𝑠𝑐 > 0,
𝑡ℎ𝑒𝑟𝑒𝑖𝑠𝑎𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡𝑛M > 0𝑠𝑢𝑐ℎ𝑡ℎ𝑎𝑡
0 < 𝑓(𝑛) < 𝑐𝑔(𝑛)
EXAMPLE: n0 = 2/c2n2 = o(n3),
Set Definition of ο-notation
CS 421 - Analysis of Algorithms 27
ο(𝑔 𝑛 ) is the set of all functions with a strictly SMALLER order of growth as 𝑔 𝑛 .
Set Definition of ο-notation
CS 421 - Analysis of Algorithms 28
ο(𝑔 𝑛 ) is the set of all functions with a SMALLER order of growth as 𝑔 𝑛 .
EXAMPLE: ο(𝑛') = 10𝑛- + 500, log 𝑛, …
10𝑛- + 500 ∈ ο(𝑛')Or put another way:
CS 421 - Analysis of Algorithms 29
ω-notationΩ-notation is like ≥.ω-notation is like >.𝑊𝑒𝑤𝑟𝑖𝑡𝑒: 𝑓 𝑛 = ω 𝑔 𝑛
𝑓𝑜𝑟𝑎𝑛𝑦𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡𝑠𝑐 > 0,
𝑡ℎ𝑒𝑟𝑒𝑖𝑠𝑎𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡𝑛M > 0𝑠𝑢𝑐ℎ𝑡ℎ𝑎𝑡
0 < 𝑐𝑔(𝑛) < 𝑓(𝑛)
CS 421 - Analysis of Algorithms 30
ω-notationΩ-notation is like ≥.ω-notation is like >.𝑊𝑒𝑤𝑟𝑖𝑡𝑒: 𝑓 𝑛 = ω 𝑔 𝑛
𝑓𝑜𝑟𝑎𝑛𝑦𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡𝑠𝑐 > 0,
𝑡ℎ𝑒𝑟𝑒𝑖𝑠𝑎𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡𝑛M > 0𝑠𝑢𝑐ℎ𝑡ℎ𝑎𝑡
0 < 𝑐𝑔(𝑛) < 𝑓(𝑛)
EXAMPLE: n0 = 1+1/cn =ω(lg n),
Set Definition of ω-notation
CS 421 - Analysis of Algorithms 31
ω(𝑓(𝑛)) is the set of all functions with a strictly LARGER order of growth as 𝑓(𝑛).
Set Definition of ω-notation
CS 421 - Analysis of Algorithms 32
ω(𝑓(𝑛)) is the set of all functions with a strictly LARGER order of growth as 𝑓(𝑛).
EXAMPLE: ω(𝑛-) = 10𝑛' + 500, 𝑛- ∗ log𝑛, …
10𝑛' ∈ ω(𝑛-)Or put another way:
Properties: Asymptotic Notation
• Transitivity• Reflexivity• Symmetry• Transposition
CS 421 - Analysis of Algorithms 33
TransitivityAssuming f(n) and g(n) are asymptotically positive:
𝑓(𝑛) = Θ(𝑔(𝑛))and
𝑔(𝑛) = Θ(ℎ(𝑛))implies
𝑓(𝑛) = Θ(ℎ(𝑛))Holds for Ο, Ω, ο, and ω relations as well.
CS 421 - Analysis of Algorithms 34
ReflexivityAssuming f(n) is asymptotically positive:
𝑓(𝑛) = Ο(𝑓(𝑛))and
𝑓(𝑛) = Ω(𝑓(𝑛))and
𝑓(𝑛) = Θ(𝑓(𝑛))
DOES NOT hold for ο and ω relations.CS 421 - Analysis of Algorithms 35
SymmetryAssuming f(n) and g(n) are asymptotically positive:
𝑓(𝑛) = Θ(𝑔(𝑛))
iff (if, and only if,)
𝑔(𝑛) = Θ(𝑓(𝑛))
CS 421 - Analysis of Algorithms 36
TransposeAssuming f(n) and g(n) are asymptotically positive:𝑓(𝑛) = Ο(𝑔(𝑛)) iff 𝑔(𝑛) = Ω(𝑓(𝑛))
and
𝑓(𝑛) = ο(𝑔(𝑛)) iff 𝑔(𝑛) = ω(𝑓(𝑛))
CS 421 - Analysis of Algorithms 37
Useful Property
𝑓1(𝑛) = Θ(𝑔1(𝑛))and
𝑓2 𝑛 = Θ(𝑔2(𝑛))implies
𝑓1 𝑛 + 𝑓2 𝑛 = Θ(max{𝑔1(𝑛), 𝑔2(𝑛)})
Holds for Ο, Ω, ο, and ω relations as well.
CS 421 - Analysis of Algorithms 41
Using Limits to Compare Growth Rates
CS 421 - Analysis of Algorithms 42
Though using Ο, Ω, ο, and ω indispensable for comparing growth rates of functions in the abstract, when comparing actual functions, convenient to
Using Limits to Compare Growth Rates
CS 421 - Analysis of Algorithms 43
limV→X
𝑓(𝑛)𝑔(𝑛)
=0 - f(n) has smaller growth rate than g(n)
c - f(n) has same growth rate as g(n)
∞ - f(n) has larger growth rate than g(n)
• first two cases ⟹𝑓 𝑛 ∈ Ο 𝑔 𝑛• last two cases ⟹𝑓 𝑛 ∈ Ω(𝑔 𝑛 )• second case ⟹𝑓 𝑛 ∈ Θ(𝑔 𝑛 )
Growth Rates (Example 2)
CS 421 - Analysis of Algorithms 49
L'Hôpital's rulehttps://en.wikipedia.org/wiki/L%27Hôpital%27s_rule
Growth Rates (Example 2)
CS 421 - Analysis of Algorithms 50
L'Hôpital's rulehttps://en.wikipedia.org/wiki/L%27Hôpital%27s_rule
Growth Rates (Example 2)
CS 421 - Analysis of Algorithms 51
Derivativeshttps://www.wyzant.com/resources/lessons/math/calculus/differentiation/list_of_derivatives
http://www.ripmat.it/mate/c/cf/cfdb.html
Growth Rates (Example 3)
CS 421 - Analysis of Algorithms 54
Stirling's approximation
https://en.wikipedia.org/wiki/Stirling%27s_approximation
Most Common Growth Rates
CS 421 - Analysis of Algorithms 57
Class Name Examples1 Constant Only used in best-case efficiencies.log n logarithmic Result of cutting problem size by a
constant factor, like Binary Searchn linear Algorithms that scan a list of size n, like
Sequential, or Linear, Searchn log n n-log-n Divide-and-Conquer algorithms, like
Merge Sort and Quick Sortn2 quadratic Efficiencies with two embedded loops,
Bubble Sort and Insertion Sortn3 cubic Efficiencies with three embedded loops,
like many linear algebra algorithms2n exponential Algorithms that generate all sub-sets of an
n-element setn! factorial Algorithms that generate all permutations
of an n-element set
Macro substitution
CS 421 - Analysis of Algorithms
Convention: A set in a formula represents an anonymous function in the set.
f(n) = n3 + O(n2) means
f(n) = n3 + h(n)for some
h(n) ∈ O(n2)
EXAMPLE:
59