evaluate algorithms

8/8/2019 Evaluate Algorithms

1/36

1

Algorithm and Complexity

Sheng Zhong

Computer Sci and Engr DeptSUNY Buffalo


2/36

2

Algorithm

The core of computer science is algorithm.

An algorithm is a finite sequence of

precise instructions for performing acomputation or for solving a problem.

Example:

Algorithm: go to NSC building; find room 210;find the 2nd row; sit in the 4th seat from the left;

calculate x=5+3.


3/36

3

Example non-algorithm

Algorithm cant be vague: go to NSC building;find room 210; find the 2nd row; sit in the 4th seat;calculate x=5+3.

The 4th seat from the left or from the right?

Doing any step of algorithm cant require

creativity or non-trivial thinking: go to NSCbuilding; find room 210; find the 2nd row; sit in

the 4th seat from the left; determine whetherP=NP and give a proof.

It requires creativity to solve the P vs. NP problem.


4/36

4

Example of non-algorithm

Algorithm cant be infinite: go to NSC

building; find room 210; find the 2nd row; sit

in the 4

th

seat from the left; calculatex=5+3, calculate x=5+4, calculate

x=5+5, ... calculate x=5+n,

Note that, if the above sequence of

instructions stops at calculatingx=5+1000,000,000, it is still an algorithmbecause it is finite.


5/36

5

Algorithm vs. program

A computer program is actually an algorithmwritten in a specific programming language. The advantage is that it can be understood and

executed by a computer.

The disadvantage is that it is hard for human beingsto understand the algorithm.

So usually algorithms are written in pseudo-codeor technical English, which is easier tounderstand. Translating an algorithm to a program is called

implementation of the algorithm in thecorresponding programming language.


6/36

6

How to write an algorithm

Below is an algorithm for finding the

maximum element in a finite sequence.

procedure max (a1, a2, , an: integers)max := a1for i:=2 to n

if max


7/36

7

Example: binary search

To demonstrate how algorithms are

designed, now we consider a problem of

searching: given a sequence of integers a1,a2, , an , where a1< a2


8/36

8

The idea

What if we start our search from the middle ofthe sequence?

Lets say we compare ai with x.

Well be very happy if we are lucky enough to find ai=x, in which case we are done.

If we find ai x, we know that all integers to the right of

ai are all greater than x, and so we only need to lookat those to the left of ai .


9/36

9

Example of binary search

Lets say we are searching for 13 in

1, 2, 3, 5, 6, 7, 8, 10, 12, 13, 15, 16, 18, 19,20, 22.

First, we compare 13 with the number inthe middle, 10.

1, 2, 3, 5, 6, 7, 8, 10, 12, 13, 15, 16, 18,

19, 20, 22.We find that 10


10/36

10


Now our remaining sequence is

12, 13, 15, 16, 18, 19, 20, 22.

Again we compare 13 with the number inthe middle, 16.

12, 13, 15, 16, 18, 19, 20, 22.

This time we find that 16>13 and hence weonly need to look at the numbers to the left.


11/36

11


Now our remaining sequence is

12, 13, 15

We compare 13 with the integer in themiddle, 13, and find that 13=13. So we are

done! This is exactly the one we need. It is

the 10th in the original sequence.


12/36

12

Binary search algorithm

Formally we can write the algorithm as follows:procedure binary search(x: integer, a1, a2, , an :increasing integers)

i:=1 {i is the left endpoint of search interval}

j:=n {j is the right endpoint of search interval}

while iam then i:=m+1

else if x


13/36

13

Other algorithm examples

The textbook has a few more examples of

algorithms like bubble sort, greedy change

making, etc. You should read them and understand howthey work.


14/36

14

Can algorithms solve all problems?

Computer algorithms are so useful. Theyhave applications in almost all aspects oftodays society. But can they solve all the problems for us?

Are they omnipotent?

The answer is NO. It suffices to observe that the set of algorithms

is countable, but the set of problems to solveis uncountable. We simply dont have enoughalgorithms to solve all the problems.


15/36

15

Example unsolvable problem

A famous problem that cant solved by algorithms is theTuring halting problem. The problem has the description of a Turing machine (you can

temporarily understand it as a computer program) as an input.

It asks whether this Turing machine will halt. (That is, whetherthe computer program will terminate after running for some time,or it will run forever.)

Unfortunately, there is no algorithm that can solve this problemfor us.

WARNING: The proof in the textbook is NOT rigorous.You should skip it. You should be able to see a rigorous proof in your theory class

(CSE 396).


16/36

16

Evaluation of algorithm

Suppose you are given an algorithm for solving acertain problem. How can you evaluate it? You can evaluate its running time (i.e., time

complexity).

You can evaluate the space it requires (i.e., spacecomplexity).

You can evaluate the amount of communications itneeds (i.e., communication complexity), the number ofrandom bits it needs,

For each of the above metric, we can evaluatethe algorithm in the average case and in theworst case. In computer science, people are normally more

interested in the worst case.


17/36

17

How to measure complexity

How do we measure complexities? For example,consider the worst-case time complexity of analgorithm.

Ideally, we should give the number of seconds thealgorithm requires in the worst case.

Unfortunately, this number can be significantlyaffected by the kind of computer you use. A supercomputer can be millions times faster than an

embedded computer. So, we count the number of basic operations (e.g.,

addition, multiplication, comparison, etc.) in stead.


18/36

18

Example complexity

What is the worst case time complexity of thealgorithm below?procedure factorial (x:integer)r:=1;for i:=1 to x

r:=r * i{r is the output}

We can easily see it requires x multiplications

(and also some other operations). Note that thisvaries with different values of input. But we dont want the complexity of an algorithm to

be a variable.


19/36

19

Complexity as a function

In general, the running time (and space,

amount of communications, ) of analgorithm vary greatly with different inputs.

Specifically, with larger inputs, we may needmore running time, more space

So, instead of using a variable to describe the

complexity, we use a function whoseargument is the length of input.


20/36

20

Example complexity

Lets look at the worst case time complexity ofthe algorithm below again.procedure factorial (x:integer)

r:=1;for i:=1 to x

r:=r * i

{r is the output}

Suppose the input has n bits. Then we needabout2n multiplications in the worst case. Here 2n is a function in n that we use to describe the

complexity.


21/36

21

Growth of function

Suppose we have three algorithms, and their worst casetime complexities are n+10, n2, and n2-1, respectively.How do they compare with each other? When n is small (say, n=2), n+10 can be larger than n2 and n2-1,

but we dont care because for smaller inputs we always havesufficient time to run the algorithm.

When n is large, n+10 is much smaller than n2 and n2-1, andthere is no significant difference between n2 and n2-1. This iswhat we really care about.

In general, when we consider complexities, we often look

at how fast a function grows, rather than look at thefunction itself. This called the asymptotic complexity of the function.


22/36

22

Big O notation

To specify the asymptotic complexity, we often use thebig O notation:

Let f and g be functions from R+ to R. We sayf(x)=O(g(x)) if there are constants C and k such that,

for all x>k, |f(x)| !!!! C|g(x)|. In the above inequality, the symbols of absolute values

can often be eliminated because we often restrict ourattention to positive valued functions. Even if a function is not positive everywhere, it should always be

positive except for a few small values of x. Here we use x as the name of argument variable. Note

that it is just n (the length of input) when we discusscomplexities.


23/36

23

Example big O

Suppose f(x)=2x3+8x. Show that f(x)=O(x3).

Proof: For each x>2, we have that

|f(x)| = 2x3+8x= 4 ( x3+2x)

< 4 x3

= 4 | x3 |.So f(x)= O(x3).


24/36

24

Equality symbol in big O

Note that f(x)=O(g(x)) is NOT a real

equality.

For example, you cant rewrite it asO(g(x))=f(x).

It does NOT mean f(x) is equal to a functioncalled O(g(x)).

It is written this way just for easiness. You must be careful with this notation.


25/36

25

Big O of polynomial function

Theorem: Suppose f(x) is a degree-n polynomial.Then f(x) = O(xn).

Proof: Since f(x) is degree-n polynomial, we can write

it as f(x)=anxn+an-1xn-1++a1x+a0.

So, for x>1 we have

|f(x)| = |anxn+an-1x

n-1++a1x+a0|

! |an| xn

+|an-1|xn-1

++|a1|x+|a0|= xn (|an| +|an-1| / x++|a1| / x

n-1+|a0| / xn)

! |xn |(|an| +|an-1|++|a1|+|a0|).

Hence, f(x)= O(xn).


26/36

26

Example polynomial big O

Suppose f(n)=1+2++n. Show that

f(n)=O(n2).

Proof. First, we note that f(n) is the sum ofan arithmetic sequence. Hence,f(n)=n+n(n-1)/2=n(n+1)/2.

Next, applying the theorem we just studied,

we get that f(n)=O(n2).


27/36

27

Big O of logarithm

It is easy to show the following fact:

For k "0, xk log x= O(xk+1).

For k "0, xk= O(xk log x). In particular, we have

log x= O(x).

1= O(log x).


28/36

28

Big O of sum

Theorem: Suppose f1 (x)=O(g1 (x)) and f2(x)=O(g2 (x)).Then

f1(x)+f2(x)=O(max(|g1(x)|,|g2(x)|))Proof: By definition of big O, we know there exist constants

k1, C1, k2, C2 , such thatfor all x>k1, |f1 (x)| !C1 |g1 (x)|;for all x>k2, |f2 (x)| !C2 |g2 (x)|.When x> both k1 and k2,|f1(x)+f2(x)| ! |f1(x)|+|f2(x)| ! C1 |g1 (x)|+ C2 |g2 (x)|

! C1 max(|g1(x)|,|g2(x)|)+ C2 max(|g1(x)|,|g2(x)|)! (C1 + C2 ) max(|g1(x)|,|g2(x)|),which means f1(x)+f2(x)=O(max(|g1(x)|,|g2(x)|)).


29/36

29

Example big O of sum

What is the big O of f(x)=x2+x2logx ?

Using the rule we just learned, f(x)= O(max(x2,x2logx))=O(x2logx).

What is the big O of f(x)=x3+x2logx ?

Using the rule we just learned, f(x)= O(max(x3,

x2logx))=O(x3).


30/36

30

Big O of product

We can also easily prove the followingtheorem:

Suppose f1 (x)=O(g1 (x)) and f2(x)=O(g2(x)). Then

f1(x)f2(x)=O(g1(x)g2(x)).

Example: f1 (x)= x2 +x+1, f2 (x)= x

3+2x+1.

So we have f1 (x)=O(x2

) and f2(x)=O(x3

).Then,

f1(x)f2(x)=O(x5).


31/36

31

Example big O

Use big O to estimate

f(x)= (xlogx+x2) (x+logx).

First, using the rule for sum, we havexlogx+x2= O(xlogx,x2)=x2.

Second, using the rule for sum, we have

x+logx= O(x,logx)=x.

Finally, using the rule for product, we have

f(x)=O(x3).


32/36

32

Big Omega notation

Intuitively, f(x)=O(g(x)) means f(x) grows

no faster than g(x) asymptotically.

But what if we want to say f(x) grows noslower than g(x) asymptotically?

We use so called big Omega notation.

We say f(x)=#$%$&''()*(%$&'+,$*$&''-

. /&012345(6)784(39%(&+(,$&':(;4(%4


33/36

33

Big Theta notation

Now we have ways to say f(x) grows no faster orno slower than g(x) asymptotically.

What if we want to say f(x) grows exactly as fast as

g(x) asymptotically? We use so called big Theta notation.

We say f(x)=>$%$&''()*(*$&'+,$%$&''(07?(

%$&'+,$*$&''-

. /&012345(6)784(39%(&+(,$39% &(@A'(07?(39%(&(@A(+(,$39% &':(;4(%4


34/36

34

Symmetry of big Theta

By the definition of big Theta, clearly we

have f(x)= >(g(x)) if and only if g(x)=

>(f(x)).

We also have that f(x)=>$%$&''()*(07?(

973B()*(*$&'+(#$%$&''(07?(%$&'+(

#$*$&''-


35/36

35

Frequently used results

In the textbook, you can find a number of

frequently used results on big O, bigOmega, and big Theta notations.

You should memorize these results, so thatyou can use them to solve problems.


36/36

36

Homework 6

Rosen 3.2: Questions 10, 20, 30.

Rosen 3.3: Questions 10, 26.

evaluate algorithms

Documents