meljun cortes algorithm limitations of algorithm power ii

8/21/2019 Meljun Cortes Algorithm Limitations of Algorithm Power II

1/12

Design and Analysis of Algorithm

Limitations of Algorithm Power *Property o f STIPage 1 of 12

TOPIC TITLE: Limitations of Algorithm Power

Speci f ic Object ives:

At the end of the topic session, the students are expected to:

Cognitive:

Understand the concept of lower-bound arguments.

Study the performance of different important algorithms usingdecision tree.

Understand P, NP and NP-complete problems.

Affective:

1. Listen to others with respect.2. Participate in class discussions actively.3. Share ideas to the class.

MATERIALS/EQUIPMENT:

o topic slideso OHP

TOPIC PREPARATION:

o Have the students research on the following: Definition of lower-bound arguments Different algorithms using decision tree Application of P, NP and NP-complete problems

o

Provide sample problems shows the limitations of algorithmpower o It is imperative for the instructor to incorporate various kinds of

teaching strategies while discussing the suggested topics. Theinstructor may use the suggested learning activities below tofacilitat e a thorough and creative discussion of the topic.

o Prepare the slides to be presented in the class.

o Prepare seatwork for the students to apply the lessons learnedunder limitations of algorithm power.

TOPIC PRESENTATION:

The topic will cover the application of limitations of algorithm power.

The following is the suggested flow of discussion for the course topic:

1. Explain the limitations of algorithm power.2. Define lower-bound then discuss the methods of establishing

lower bounds.3. Explain decision trees and sorting algorithm.4. Discuss the applications of P, NP, and NP-complete problems.


2/12



Limitations of Algorithm PowerPage 1 of 25

Limitationsof algorithm Power

DesignandAnalysisof Algorithm

* Property of STI

Page1of 26

The following are the topics to be discussed under

Limitations of algorithm power

Definition of lower-bound arguments

Different algorithms using decision tree

Application of P, NP and NP-complete

problems

Limitations of Algorithm Power

The following are the topics to be discussed under Limitations ofalgorithm power:

Definition of lower-bound arguments

Different algorithms using decision tree Application of P, NP, and NP-complete problems

[Limitations of Algorithm Power, Page 1 of 25]

What are Limitations of AlgorithmPower?Page 2 of 25



* Property of STI Page2of 26

A reasonable evaluation of algorithms as problem-

solving tools is inevitable.

These algorithms are considered as powerful

instruments, particularly when these are

processed by modern computers.

Sometimes there are problems that cannot be

solved by an algorithm or the problems seem not

to have efficient solutions.

What are Limitations of Algorithm Power?

In the previous topics, we have studied the different algorithm designtechniques for solving various problem types. A reasonable evaluationof algorithms as problem-solving tools is inevitable. These algorithmsare considered as powerful instruments, particularly when these areprocessed by modern computers. However, the power of algorithms isnot infinite, and its limits are the major concerns. Sometimes there areproblems that cannot be solved by an algorithm or the problems seemnot to have efficient solutions. In fact, there are problems that can besolved algorithmically, but not in polynomial time. And even when aproblem can be solved in polynomial time by some algorithms, there are

usually lower bounds on their efficiency.

[What are Limitations of Algorithm Power?, Page 2 of 25]

Lower –Bound ArgumentsPage 3 of 25



* Property of STI

Page3of 26

Lower bound :is a comparison-based algorithm

where the behavior of the algorithm is based only

on the comparison between elements

[www.cs.toronto.edu].

The lower bound deals with the complexity of a

problem instead of the algorithm.

Basically we have to prove that no algorithm , no

matter how complex, can do better than our

bound

Lower –Bound Arguments

Lower bound is a comparison-based algorithm where the behavior of the

algorithm is based only on the comparison between elements[www.cs.toronto.edu].

The lower bound deals with the complexity of a problem instead of thealgorithm. Basically we have to prove that no algorithm, no matter howcomplex, can do better than our bound. This is actually difficult, we saythat f(n) is a lower bound on the complexity of a problem P if, for everyalgorithm A to solve P, and every n, there exists some input I of size nsuch that A uses at least f(n) steps on input I.

Examples:

number of comparisons needed to find the largest element in aset of n numbers

number of comparisons needed to sort an array of size n number of comparisons necessary for searching in a sorted


3/12







Examples:

number of comparisons needed to find the largest

element in a set of n numbers

number of comparisons needed to sort an array of

size n

number of comparisons necessary for searching in

a sorted array

number of multiplications needed to multiply two

n-by-n matrices




* Property of STI

Page5of 25

Two ways to examine the algorithm efficiency:

Establish the asymptotic efficiency class similar to

the worst case

Single out where this class stands with respect to

the hierarchy of efficiency class

array

number of multiplications needed to multiply two n-by-n matrices

Normally, we can examine the efficiency of an algorithm in two waysthrough establishing its asymptotic efficiency class similar to the worstcase and single out where this class stands with respect to the hierarchy

of efficiency classes. For instance, a selection sort algorithm where theefficiency is quadratic and considered as the fastest algorithm whereasthe Tower of Hanoi algorithm is very slow since its efficiency isexponential. On the other hand, the comparison is similar to the well-known comparison of grapes from melons in view of the fact that thesetwo algorithms solve exactly dissimilar problems. Another alternative isto ask how efficient a particular algorithm is with respect to the otheralgorithms for the same problem. As a result, selection sort has to beconsidered slow because there are O(n log n) sorting algorithms and theTower of Hanoi algorithm, on the other hand, turns out to be the fastestpossible for the problem it solves.

As soon as we want to find out the efficiency of an algorithm with

respect to other algorithms for the same problem, it is required to knowthe best possible efficiency any algorithm solving the problem may have.The lower bound will be able to inform us how much improvement wecan expect to reach in our searching technique for a better algorithm ofthe problem. If in case, a bound is tight, for instance, we already knowan algorithm in the same efficiency class as the lower bound, we canhope for a constant-factor improvement at best. If there is a gapbetween the efficiency of the fastest algorithm and the best lower boundknown, the door for possible improvement remains open. Moreover,faster algorithms matching the lower bound exist or a better lower boundcould be permitted.

[Lower-Bound Arguments, Pages 3-5 of 25]

Methods for Establishing LowerBoundsPage 6 of 25



* Property of STI

Page6of 25

Trivial lower bound: is based on counting the

number of items that must be processed in input

and generated as output [LEV07].

Examples:

finding max element --n steps or n/2

comparisons

polynomial evaluation

sorting

element uniqueness

Hamiltonian circuit existence

Conclusions:

may and may not be useful

be careful in deciding how many elements

must be processed

Methods for Establishing Lower Bounds

The following are the different methods in establishing lower bounds:

Trivial lower bound is based on counting the number of items

that must be processed in input and generated as output[LEV07].

Examples:

finding max element -- n steps or n/2 comparisons

polynomial evaluation

sorting

element uniqueness


Conclusions:

may and may not be useful

be careful in deciding how many elements must beprocessed


4/12







Information-theoretic arguments decision trees) :

seek to establish a lower bound based on the

amount of information it has to produce [LEV07].

Example:

Deducing a positive integer between 1 and n

selected by somebody by asking that person

questions with yes/no answers.

Conclusions:

The approach is related to information theory

because it has proved to be useful for finding

the information-theoretic bounds in different

problem types concerning comparisons, such

as sorting and searching.

Its core concept can be apprehended more

precisely through decision trees.

Information-theoretic arguments (decision trees) seek toestablish a lower bound based on the amount of information ithas to produce [LEV07].

Examples:

Deducing a positive integer between 1 and n selected

by somebody by asking that person questions withyes/no answers.

Conclusions:

The approach is related to information theory because ithas proved to be useful for finding the information-theoretic bounds in different problem types concerningcomparisons, such as sorting and searching.

Its core concept can be apprehended more preciselythrough decision trees.





Adversary arguments: Malevolent simply means

that it will change a problem to force algorithm

down worst path and we can say that it is fair

because we must remain consistent with the work

that is already done for selection problems.

Examples:

Three-Card Monte

n-Card Monte

Finding Patterns in Bit Strings

Evasive Graph Properties

Conclusions:

The adversary should be thought of as a very

powerful, clever being that is trying to make

your algorithm run as slowly as possible.

The adversary cannot "read the algorithm's

mind", but it can try to be prepared for

anything the algorithm does.

Adversary arguments are applied to selection problems. Theseassume a malevolent, but fair adversary. Malevolent simplymeans that it will change a problem to force algorithm downworst path and we can say that it is fair because we mustremain consistent with the work that is already done for selectionproblems.

Examples:

Three-Card Monte

n-Card Monte

Finding Patterns in Bit Strings


Conclusions:

The adversary should be thought of as a very powerful,clever being that is trying to make your algorithm run asslowly as possible. The adversary cannot "read thealgorithm's mind", but it can try to be prepared foranything the algorithm does.




* Property of STI

Page9of 25

Problem reduction: proves a lower bound by

transforming a problem that already has an

established lower bound into the new problem.

Examples:

Reduction to linear programming

Algorithmic problem solving

Reduction to graph problems


Problem reduction proves a lower bound by transforming aproblem that already has an established lower bound into thenew problem.

In fact, we have already discussed the problem reduction topic,which simply states that the problem Q has a known lowerbound, and problem Q can be transformed into problem P, thenthe lower bound for P must be the same as the lower bound forQ.

Examples:

Reduction to linear programming

Algorithmic problem solving

Reduction to graph problems


Conclusions:We must prove that an arbitrary instance of problem Q


5/12






* Property of STI

Page10of 25

Conclusion:

We must prove that an arbitrary instance of

problem Q can be transformed in a logically

efficient manner to an instance of problem P.

As a result, any algorithm solving P would solve Q

as well. Therefore, the lower bound for Q will also

be the lower bound for P.

The figure below are the list of problems used for

establishing lower bounds by problem reduction:

can be transformed in a logically efficient manner to aninstance of problem P. As a result, any algorithmsolving P would solve Q as well. Therefore, the lowerbound for Q will also be the lower bound for P.

The figure below are the list of problems used forestablishing lower bounds by problem reduction:

Figure 16.1 List of problems commonly used for establishing lower boundsthrough problem reduction

[Methods for Establishing Lower Bounds, Pages 6-10 of 25 ]

Decision TreesPage 11 of 25



* Property of STI

Page11of 25

Decision tree : is a convenient model of algorithms

involving comparisons in which internal nodes

represent comparisons and leaves representoutcomes (or input cases).

The figure below represents the decision tree of

an algorithm for finding a minimum value of three

numbers:

Decision Trees

Decision tree is a convenient model of algorithms involving comparisonsin which internal nodes represent comparisons and leaves representoutcomes (or input cases).

The figure below represents the decision tree of an algorithm for findinga minimum value of three numbers.

Figure 16.2 Sample decision tree for finding a minimum value of three numbers

In the figure, every internal node of a binary decision tree signifies a keycomparison specified in the node, like k < k’, at the same time its rightsubtree have the information about the succeeding comparisons made ifk < k’. On the other hand, its right subtree does the same for the case ofk > k’. Each leaf represents a probable result of the algorithm’s run on

some input size n. Always remember that the number of leaves can begreater than the number of results since, for some algorithms, the same


6/12



result can arrived at a different chain of comparisons. In addition, asignificant point is that the number of leaves must be at least as large asthe number of possible results. Therefore, the algorithm’s work on aparticular input size n can be located by a path from the root to a leaf inits decision tree, and the number of comparisons made by the algorithmon such a run is equal to the number of edges in this path.

Consequently, the amount of comparisons in the worst case is equal tothe height of the algorithms decision tree.

The main idea behind this model lays in the study that a tree with agiven number of leaves, which is specified by the number of probableresults, has to be tall enough to possess that various leaves.Specifically, it is not difficult to prove that for any binary tree with l leavesand height h: h ≥ [log2l ].

[Decision Trees, Page 11 of 25]

Decision Trees forSorting Algorithm

Page 12 of 25



* Property of STI

Page12of 25

By studying the properties of decision trees for

comparison-based sorting algorithms, we can

derive important lower bounds on time efficiencies

of such algorithms.

Consider the following figure:

Decision Trees for Sorting Algorithm

In comparing two elements in a sorting algorithm, we can simply say thatthe only results can be “yes” or “no”. Thr ough this comparison, thesorting algorithm may perform some calculations that are not concernedwith and will ultimately perform another comparison between two otherelements of input series, which will again result to have two outcomes.For these reasons, we can use the decision tree as a tool to represent a

comparison-based sorting algorithm.

To further understand the application of decision tree, let us examine ata specific decision tree that sorts three elements {A,B,C}, where theinternal nodes denote comparisons of the two elements in the node. Onthe other hand, the leaf nodes represent the outcome of the sorting

process. Therefore, each edge is labeled with the outcome of thecomparison of the node above it.

By studying the properties of decision trees for comparison-basedsorting algorithms, we can derive important lower bounds on timeefficiencies of such algorithms. Consider the following figure:

Figure 16.3 Sample decision tree for finding a minimum value of three numbers


7/12



In the given figure, we can interpret an outcome of a sorting algorithm asfinding a permutation of the element indices of an input list that puts thelist’s elements in ascending order. For instance, in the outcome A


8/12



P, NP and NP-CompleteProblemsPage 15 of 25



* Property of STI

Page15of 25

An algorithm solves a problem in polynomial timeif its worst-case time efficiency belongs to O(p(n))

time, where p(n) is a polynomial of problem’s input

size n [LEV07].

Tractable : problems that can be solved in

polynomial time.

Intractable : problems that cannot be solved in

polynomial time.

P, NP and NP-Complete Problems

In studying computational complexity of problems, the primaryapprehension of computer scientists and other computing professionalsis whether a given problem can be solved in polynomial time by somealgorithm.

An algorithm solves a problem in polynomial time if its worst-case timeefficiency belongs to O( p(n)) time, where p(n) is a polynomial ofproblem’s input size n [LEV07].

Problems that can be solved in polynomial time are referred to astractable. While problems that cannot be solved in polynomial time areknown as intractable.

P, NP and NP-CompleteProblems

Page 16 of 25



* Property of STI

Page16of 25

Reasons for drawing the intractability line :

We cannot solve arbitrary instances of

intractable problems in a reasonable amount

of time unless such instances are ve ry small.

Although there might be a huge difference

between the running time in O(p(n)) for

polynomials of drastically different degrees,

there are very few useful polynomial time

algorithm with the degree of polynomial higher

than three.

Polynomial functions posses many convenient

properties, in particular both the sum and

composition of two polynomials are always

polynomials, too.

The choice of this class led to a development

of an extensive theory called computational

complexity.

There are reasons for drawing the intractability line and these are asfollows:

We cannot solve arbitrary instances of intractable problems in areasonable amount of time unless such instances are verysmall.

Although there might be a huge difference between the runningtime in O(p(n)) for polynomials of drastically different degrees,there are very few useful polynomial time algorithm with thedegree of polynomial higher than three.

Polynomial functions posses many convenient properties, inparticular both the sum and composition of two polynomials arealways polynomials, too.

The choice of this class led to a development of an extensivetheory called computational complexity.

P, NP and NP-CompleteProblemsPages 17-18 of 25



* Property of STI

Page17of 25

Class P: is a class of decision problems that can

be solved in polynomial time by deterministic

algorithms. This class of problems is called

polynomial [LEV07].

The restriction of P to decision problems can be

justified by the following:

It is sensible to exclude problems not solvable

in polynomial time because of their

exponentially large output.

Different important problems that are not

decision problems that are easier to study.

Class P is a class of decision problems that can be solved in polynomialtime by deterministic algorithms. This class of problems is calledpolynomial [LEV07].

The restriction of P to decision problems can be justified by thefollowing:

1. It is sensible to exclude problems not solvable in polynomialtime because of their exponentially large output.

2. Different important problems that are not decision problems thatare easier to study.

There are many important problems, however, for which polynomial timealgorithm has been found, or has the impossibility of such an algorithmbeen proved. The class is monograph by M. Garey and D. Johnson[Gar79] contains a list of several problems from different areas ofcomputer science, mathematics and operations research.


9/12





* Property of STI

Page18of 25

There are many important problems showing the

polynomial time algorithm:

Hamilton circuit

Traveling salesman

Knapsack problem

Partition problem Bin packing

Graph coloring

Integer linear programming

Examples of polynomial time problems are:1. Hamilton circuit2. Traveling salesman3. Knapsack problem4. Partition problem5. Bin packing

6. Graph coloring7. Integer linear programming

P, NP and NP-CompleteProblems

Pages 19-20 of 25



* Property of STI

Page19of 25

NP nondeterministic polynomial):the class of

decision problems whose proposed solutions can

be verified in polynomial time, which is solvable by

a nondeterministic polynomial algorithm.

A nondeterministic polynomial algorithm is an

abstract two-stage procedure that:

generates a solution of the problem (on some

input) by guessing

checks whether this solution is correct in

polynomial time




NP nondeterministic polynomial):the class of

decision problems whose proposed solutions can

be verified in polynomial time, which is solvable by

a nondeterministic polynomial algorithm.

A nondeterministic polynomial algorithm is an

abstract two-stage procedure that:

generates a solution of the problem (on someinput) by guessing

checks whether this solution is correct in

polynomial time

Sample NP or Nondeterministic algorithm

problems:

Guess a truth assignment

Substitute the values into CNF formula to see

if it evaluates to true

Class NP

NP (nondeterministic polynomial): the class of decision problems whoseproposed solutions can be verified in polynomial time, which is solvableby a nondeterministic polynomial algorithm.

A nondeterministic polynomial algorithm is an abstract two-stageprocedure that:

generates a solution of the problem (on some input) by guessing

checks whether this solution is correct in polynomial time

We say that nondeterministic algorithm solves a decision problem if andonly if for every yes instance of the problem, it returns yes on someexecution.

By definition, it solves the problem if it’s capable of generating andverifying a solution on one of its tries. This definition will lead to thedevelopment of the rich theory called “computational complexity” like theCNF satisfiability problem.

Problem: Is a Boolean expression in its conjunctive normal form (CNF)satisfiable, for instance, a question asking if there are values of itsvariables that makes it true?

This problem is in NP. Nondeterministic algorithm:

Guess a truth assignment

Substitute the values into CNF formula to see if it evaluates to

true


10/12



P, NP and NP-CompleteProblemsPages 21-22 of 25



* Property of STI

Page21of 25

Other Problems in NP

Class NP :is the class of decision problems that

can be solved by nondeterministic polynomial

algorithms. This class of problems is called

nondeterministic polynomial [LEV07].

All the problems in P can also be solved in this

manner (but no guessing is necessary), so we

have:

• P NP



* Property of STI

Page22of 25

NP sample problems also includes:


Partition problem: Is it possible to partition a

set of n integers into two disjoint subsets with

the same sum?

Decision versions of TSP, knapsack problem,

graph coloring, and many other combinatorial

optimization problems.

Other Problems in NP

Class NP is the class of decision problems that can be solved bynondeterministic polynomial algorithms. This class of problems is callednondeterministic polynomial [LEV07].

All the problems in P can also be solved in this manner (but no guessingis necessary), so we have:

P NP

Therefore, we can use the deterministic polynomial time algorithm thatsolves it in the verification stage of a nondeterministic algorithm thatsimply ignores string generated in its nondeterministic (“guessing”)stage.

NP sample problems also includes:


Partition problem: Is it possible to partition a set of n integers

into two disjoint subsets with the same sum?Decision versions of TSP, knapsack problem, graph coloring,and many other combinatorial optimization problems.

This will lead to the most important open question of theoreticalcomputer science: Is P a proper subset of NP, or are these two classesthe same? We can put this symbolically as:

P = NP ?

Note that P = NP would imply that each of many difficult combinatorialdecision problems can be solved by a polynomial time algorithm.

Although computer scientists have failed to find such algorithms despite

their persistent efforts over many years. Moreover, many well-knowndecision problems are known to be “NP-complete” that seems to castmore doubts on the possibility that P = NP.

[P, NP and NP-Complete Problems, Pages 15-22 of 25]

NP-Complete ProblemsPages 23-25 of 25



* Property of STI

Page23of 25

NP-complete problem: i s a p robl em i nNP tha t i s

a s d if fi cu lt a s a ny other p robl em i n thi s c la ss ,

because by definition, any problem in NP can be

reduced toit inpolynomial time. A decision problem D1 is said to be polynomially

reducible to a decision problem D2 if there exists a

function t that transforms instances of D1 to

instances of D2 such that:

t maps all yes instances of D 1 to yes instances

of D2 and all no instances of D1 to all no

instances of D2

t is computable by a polynomial time

algorithm.

A decision problem D is NP -complete if it is as

hard as any problem in NP , i.e.,

D is in NP

every problem in NP is polynomial-time reducible

to D

NP -Complete Problems

Informally, an NP-complete problem is a problem in NP that is as difficultas any other problem in this class, because by definition, any problem inNP can be reduced to it in polynomial time.

A decision problem D1 is said to be polynomially reducible to a decision

problem D2 if there exists a function t that transforms instances of D 1 toinstances of D2 such that:

1. t maps all yes instances of D1 to yes instances of D2 and all noinstances of D1 to all no instances of D2

2. t is computable by a polynomial time algorithm.

A decision problem D is NP -complete if it is as hard as any problem inNP , i.e.,

D is in NP

every problem in NP is polynomial-time reducible to D

This definition immediately implies that if D2 is solvable, then D1 can alsobe solved in polynomial time, as illustrated in the following figure.


11/12





* Property of STI

Page24of 25

This definition immediately implies that if a

problem D is polynomially reducible to some

problem D that can be solved in polynomial time,

then problem D.

known

NP -complete

problem

NP problems

candidate

for NP -

completeness



* Property of STI

Page25of 25

Other NP -complete problems obtained through

polynomial-time reductions from a known NP -

complete problem, as illustrated in the figure

below.

known

NP -complete

problem

NP problems

candidate

for NP -

completeness

NP -complete

problem

NP problems

Figure 16.5 List of problems commonly used for establishing lower bounds

through problem

Other NP -complete problems obtained through polynomial-timereductions from a known NP -complete problem, as illustrated in the

figure below.

known

NP -complete

problem

NP problems

candidate

for NP -

completeness

Figure 16.6 List of problems commonly used for establishing lower bounds

through problem

Examples: TSP, knapsack, partition, graph-coloring and hundreds ofother problems of combinatorial nature.

[NP-Complete Problems, Pages 23-25 of 25]

GENERALIZATION:

o Sometimes there are problems that cannot be solved by analgorithm or problems seem not to have efficient solutions itseems not to have efficient solutions. In fact, there areproblems than can be solved algorithmically but not inpolynomial time.

o Lower bound is a comparison-based algorithm is an algorithmwhere the behavior of the algorithm is based only on thecomparisons between elements.

o Decision tree is a convenient model of algorithms involvingcomparisons in which internal nodes represent the comparisons

and leaves represent outcomes or input cases.o In the studying computational complexity of problems, the

primary apprehension of computer scientists and other


12/12



computing professionals is whether a given problem can besolved in polynomial time by some algorithm.

REFERENCES:

www.scuec.edu.cn/jsj/scuec_cs/jinping/algorithm/.../Lecture17.p

pt Anany Levitin,(2007), The design and analysis of algorithm (2nd

ed.), Pearson Education Inc.

meljun cortes algorithm limitations of algorithm power ii

Documents