design and analysis of algorithms 1

8/6/2019 Design and Analysis of Algorithms 1

1/29

.

.. ..

.Design and Analysis of Algorithms

Lecture 1: Introduction

Ahmed Khademzadehhttp://khademzadeh.mshdiau.ac.ir

Azad University of Mashhad

The slides are borrowed from Ke (Kevin) Yi. Thanks to him for his benevolence.

Lecture 1: Introduction Design and Analysis of Algorithms
http://-/?-


2/29

. Outline

What is this course about?

What are algorithms?

What does it mean to analyze an algorithm?

Comparing time complexity

Thoughts on algorithm design



3/29

. What is this course about?.Example (Chain Matrix Multiplication).

.. .

.

.

A = C =[

1 1 0 1].

B = D =

10

11

.

Want: ABCD =?

Method 1: (AB)(CD)

Method 2: A((BC)D)

Method 1 is much more efficient than Method 2.(Expand the expression on board)



4/29

. What is this course about?

There is usually more than one algorithm for solving aproblem.

Some algorithms are more efficient than others.

We want the most efficient algorithm.



5/29

. What is this course about?

If we a number of alternative algorithms for solving a problem,how do we know which is the most efficient?

To do so, we need to analyze each of them to determine itsefficiency.

Of course, we must also make sure the algorithm is correct.



6/29

. What is this course about?In this course, we will discuss fundamental techniques for:...1 Proving the correctness of algorithms,...2 Analyzing the running times of algorithms,...3 Designing efficient algorithms,...4 Showing efficient algorithms do not exist for some problemsNotes:

Analysis and design go hand-in-hand:

By analyzing the running times of algorithms, we

will know how to design fast algorithms.

In Data Structure course, you have learned techniques for 1,2, 3 w.r.t sorting and searching. In this course, we will discussthem in the contexts of other problems.



7/29

. What is this course about?Differences between Data Sturcture and Algorithm Design Course:.

Data Sturcture.

.. .

.

.

The name is Data Sturcture

Simpler problemsYou learn concrete problems:sorting, searching (binarytree), etc.

You know how to solvethese concrete problems.

.Algorithm Design Analysis.

.. .

.

.

The name is AlgorithmDesign Analysis

Harder problems

You learn general techniquesvia concrete problems:divide-and-conquer, dynamicprogramming, etc.

You can solve many newproblems using thesetechniques.



8/29

. Outline



What does it mean to analyze an algorithm?Comparing time complexity



C


9/29

. Computational Problem.Definition... .

..

A computational problem is a specification of the desiredinput-output relationship

.Example (Computational Problem).

.. .

.

.

Sorting

Input: Sequence of n numbers a1, , an

Output: Permutation (reordering)

a

1, a

2, , a

n

such that a1 a

2 a

n


I


10/29

. Instance

.Definition... .

..A problem instance any valid input to the problem.

.Example (Instance of the Sorting Problem).

.. ..

.8, 3, 6, 7, 1, 2, 9


Al i h


11/29

. Algorithm

.Definition.

.. .

.

.

An algorithm is a well defined computational procedure thattransforms inputs into outputs, achieving the desired input-output

relationship.Definition... .

..

A correct algorithm halts with the correct output for every inputinstance. We can then say that the algorithm solves the problem


E l I ti S t


12/29

. Example: Insertion SortPseudocode:

Input: A[1 . . . n] is an array of numbers

for j=2 to n dokey = A[j];i = j-1;while i 1 and A[i]> key do

A[i+1] = A[i];

i- -;endA[i+1] = key;

end

key

Sorted UnsortedWhere in the sorted part to put key?


H D It W k?


13/29

. How Does It Work?An incremental approach: To sort a given array of length n,

at the ith step it sorts the array of the first i items by makinguse of the sorted array of the first i 1 items in the (i 1)thstep

.Example

.

.. .

.

.

Sort A = 6, 3, 2, 4, 5 with insertion sort

Step 1: 6, 3, 2, 4, 5

Step 2: 3, 6, 2, 4, 5

Step 3: 2, 3, 6, 4, 5

Step 4: 2, 3, 4, 6, 5

Step 5: 2, 3, 4, 5, 6


O tli


14/29

. Outline






Analyzing Algorithms


15/29

. Analyzing AlgorithmsPredict resource utilization

...1 Running time (time complexity) focus of this course...2 Memory (space complexity)Running time: the number of primitive operations (e.g., addition,multiplication, comparisons) used to solve the problem

Depends on problem instance: often we find an upper bound:T(input size) and use asymptotic notations (big-Oh)

To make your life easier!

Growth rate much more important than constants in big-Oh.

Input size n: rigorous definition given latersorting: number of items to be sorted

graphs: number of vertices and edges


Three Cases of Analysis: I


16/29

. Three Cases of Analysis: IBest Case: An instance for a given size n that results in the fastestpossible running time.

.Example (Insertion sort).

.. .

.

.

A[1] A[2] A[3] A[n]

The number of comparisons needed is equal to

1 + 1 + 1 + + 1 n1

= n 1 = O(n)

key

Sorted Unsortedkey is compared to only the element right before it.


Three Cases of Analysis: II


17/29

. Three Cases of Analysis: IIWorst Case: An instance for a given size n that results in theslowest possible running time..Example (Insertion sort).

.. .

.

.

A[1] A[2] A[3] A[n]

The number of comparisons needed is equal to

1 + 2 + + (n 1) =n(n 1)

2= O(n2)

key

Sorted Unsortedkey is compared to everything element before it.


Three Cases of Analysis: III


18/29

. Three Cases of Analysis: IIIAverage Case: T running time averaged over all possible instancesfor the given size, assuming some probability distribution on theinstances..Example (Insertion sort)... .

.

.(n2

), assuming that each of the n! instances is equally likely(uniform distribution).

key

Sorted UnsortedOn average, key is compared to half of the elements before it.


Three Cases of Analysis


19/29

. Three Cases of Analysis

Best case: Clearly the worst

Worst case: Commonly used, will also be used in this course

Gives a running time guarantee no matter what the input isFair comparison among different algorithms

Average case: Used sometimes

Need to assume some distribution: real-world inputs areseldom uniformly random!Analysis is complicatedWill seldom be used in this course


Outline


20/29

. Outline






Example


21/29

. Example

n

T(n)

Algorithm 1

Algorithm 2

Algorithm 2 is clearly superiorT(n) for Algorithm 1 is O(n3)

T(n) for Algorithm 2 is O(n2)

Since n3 grows much more rapidly, we expect Algorithm 1 totake much more time than Algorithm 2 when n increases


Growth Rate


22/29

. Growth RateGiven algorithms A and B with running times TA(n) and TB(n):

If limnTA(n)TB(n)

= 0, we say TA has a lower asymptotic growth

rate than TB. (Thus, A runs faster than B for large n.)

If limnTA(n)

TB(n) = , we say TA has a higher asymptotic growthrate than TB. (Thus, A runs slower than B for large n.)

If 0 < limnTA(n)

TB(n)


23/29

. Big OhAsymptotic upper bound.Definition (Big-Oh)... .

.

.f(n) = O(g(n)): There exists constant c> 0 and n0 such thatf(n)c g(n) for n n0

When estimating the growth rate of T(n) using big-Oh:

ignore the low order terms

ignore the constant coefficient of the most significant term.Example... .

.

.Algorithm 1 runs in O(n3) time and Algorithm 2 runs in O(n2)

time.Example... .

..

It is also OK to say Algorithm 2 runs in O(n3) time, but we usuallywant to be as tight as possible



24/29

.Example.

.. .

.

.

n

2

/2 3n = O(n

2

)1 + 4n = O(n)

log10 n =log2 n

log2 10= O(log2 n) = O(log n)

sin n = O(1), 10 = O(1), 1010 = O(1)

ni=1 i2 n n2 = O(n3)n

i=1 i n n = O(n2)

210n is not O(2n)

log(n!) = log(n) + log(n 1) + + log 1 = O(n log n)

ni=1 1i = O(log n) (on board)


Big Omega and Big Theta


25/29

. g g gAsymptotic lower bound.Definition (big-Omega)... .

.

.f(n) = (g(n)): There exists constant c> 0 and n0 such thatf(n) c g(n) for n n0.

Asymptotic tight bound.Definition (big-Theta)... .

. .f(n) = (g(n)): f(n) = O(g(n)) and f(n) = (g(n)).Example.

.

. .

.

.

n2

/2 3n = (n2

)log(n!) = log(n) + log(n 1) + + log 1 log(n) + log(n 1) + + log(n/2) n/2 log(n/2)= n/2 (log n 1) = (n log n).


Big Omega and Big Theta


26/29

. g g g

.Example.

.. .

.

.

If Algorithm 1 runs in O(n3) time (we can show that any instanceof size n, the algorithm takes at most O(n3) time), and

Algorithm 1 runs in (n3

) time (there is an instance of size n onwhich the algorithm spends (n3) time),

then we say Algorithm 1 runs in (n3) time.

Usually (and in this course), it is sufficient to show only upperbounds (big-Oh).


Outline


27/29

.






Some Thoughts on Algorithm Design


28/29

.

Algorithm Design, as taught in this class, is mainly aboutdesigning algorithms that have small big-Oh running times

As n gets larger and larger, O(n log n) algorithms will runfaster than O(n2) ones and O(n) algorithms will beatO(n log n) ones

Good algorithm design & analysis allows you to identify thehard parts of your problem and deal with them effectively

Too often, programmers try to solve problems using bruteforce techniques and end up with slow complicated code!

A few hours of abstract thought devoted to algorithm designoften results in faster, simpler, and more general solutions.


Algorithm Tuning


29/29

.

After algorithm design one can continue on to Algorithm tuning

concentrate on improving algorithms by cutting down on theconstants in the big O() bounds

needs a good understanding of both algorithm designprinciples and efficient use of data structures

In this course we will not go further into algorithm tuning

For a good introduction, see chapter 9 in ProgrammingPearls, 2nd ed by Jon Bentley


design and analysis of algorithms 1

Documents