caching parallel computational models other topics in algorithms wednesday, august 13 th

1

Caching

Parallel Computational Models

Other Topics in Algorithms

Wednesday, August 13th

2

Announcements

1. PS#6 due tonight at midnight

2. Winners of the Competition:

1. Shir Aharon

2. Rasoul Kabirzadeh

3. Alice Yeh & Marie Feng

3. Extra Office Hours on Thursday & Friday

Semih: Th 10am-12pm (Gates 424)

Billy: Friday 1pm-5pm (Gates B24)

Mike: Th 3pm-5pm (Gates B24)

Yiming: Fr: 5pm-7pm (Gates B24)

3

Outline For Today

1. Caching

2. Other Algorithms & Algorithmic Techniques Beyond CS

161

1. Parallel Algorithms (Finding Min, Bellman-Ford)

2. Linear Programming

3. Other Topics & Classes

4

Alan Turing: Father of Computer Science

5

Computer Science

Studies the powers of machines.

Fundamental Question CS asks:

What is “computable” by machines?

Turing, along with Church and Godel, was the

person who made computation something we

can mathematically study.

6

Turing’s Answer To What Computation Is (1936)“On Computable Numbers, with an Application

to the Entscheidungsproblem”:

“We may compare a man in the process of

computing a real number to a machine which

is only capable of a finite number of

conditions”

7

Turing’s Answer To What Computation IsComputing is normally done by writing certain symbols on paper. We may suppose this paper is divided into squares like a child's arithmetic

book. The behavior of the computer at any moment is determined by the symbols which he is observing, and his " state of mind " at

that moment. We may suppose that there is a bound B to the number of symbols or squares

which the computer can observe at one moment. If he wishes to observe more, he must use successive observations. We will also suppose that the number of states of mind which need be taken into account is

finite.

8

Turing’s Answer To What Computation Is

Let us imagine the operations performed by

the computer to be split up into "simple

operations" which are so elementary that it is

not easy to imagine them further divided.

Every such operation consists of some change

of the physical system consisting of the

computer and his tape. … We may suppose

that in a simple operation not more than one

symbol is altered.

9

Turing’s Answer To What Computation IsBesides these changes of symbols, the simple

operations must include changes to the

observed squares. … I think it is reasonable to

suppose that they can only be squares whose

distance from the closest of the immediately

previously observed squares does not exceed

a certain fixed amount.

10

The Turing Machine

It’s paradoxical that as humans in our quest to understand what machines can do, we have been studying an abstract machine that in

essence imitates a human being.

11

Church-Turing Thesis

Central Dogma of Computer Science:

**Whatever is computable is

computable by the Turing machine.**

There is no proof of this claim.

12

Turing In Defense Of His Claim:

“All arguments which can be given are bound to

be, fundamentally, appeals to intuition, and for

this reason rather unsatisfactory mathematically.

The arguments which I shall use are of three

kinds.

(a) A direct appeal to intuition.

(b) A proof of the equivalence of two definitions

(in case the new definition has a greater intuitive

appeal).

(c) Giving examples of large classes of numbers

which are computable.”

13

Algorithm = Turing Machine

When we say there is an algorithm computing

shortest paths of a graph in O(mlog(n)) times

we really mean:

There is a Turing Machine that computes the

shortest paths of a graph in O(mlog(n))

operations.

14

Turing Machine Is A Very Powerful MachineCS tries to understand the limits of TM. We limit/extend TM and try to understand what can be computed by it. Limit the # times it’s head is allowed to move left/right

and it changes states to poly-time. => poly-time algs What if the machine had access to a random source

=> randomized algorithms. What if there were multiple heads on the tape =>

parallel algorithms Limit the length of its tape. => space-efficient algs What if the head was only allowed to move right =>

streaming algorithms …

15

Outline For Today

1. Caching


161




16

Online Algorithms Takes as input a possibly infinite stream.

At each point in time t make a decision based on

what has been seen so far

but without knowing the rest of the input

Type of Optimality Analysis: Competitive Ratio

“Worst” (Cost of online algorithm)/(Cost of OPT)

ratios against any input stream

Where OPT is the best solution possible if we

knew the entire input in advance

17

Caching

Slow Disk

Fast CachePage1

……

Pagek

… … … R3 R2 R1O.w (miss),

send request to disk, put the

page into cache.

Q: Which page to evict?

If page is in cache (hit) reply directly from cache

18

Caching

Input: N pages in disk, and stream of infinite page

requests.

Online Algorithm: Decide which page to evict from

cache when it’s full and there’s a miss.

Goal: minimize the number of misses.

Idea: LRU: Remove the Least Recently Used

page

19

LRU with k = 3

4 1 2 1 5 3 4 4 1 1 3 2 4 5 1

LRU

miss

20

LRU with k = 3

4 1 2 1 5 3 4 4 1 1 3 2 4 5 1

LRU

4

21

LRU with k = 3

4 1 2 1 5 3 4 4 1 1 3 2 4 5 1

LRU

4

miss

22

LRU with k = 3

4 1 2 1 5 3 4 4 1 1 3 2 4 5 1

LRU

4

1

23

LRU with k = 3

4 1 2 1 5 3 4 4 1 1 3 2 4 5 1

LRU

4

1

miss

24

LRU with k = 3

4 1 2 1 5 3 4 4 1 1 3 2 4 5 1

LRU

4

1

2

25

LRU with k = 3

4 1 2 1 5 3 4 4 1 1 3 2 4 5 1

LRU

4

1

2

hit

26

LRU with k = 3

4 1 2 1 5 3 4 4 1 1 3 2 4 5 1

LRU

4

1

2

miss

27

LRU with k = 3

4 1 2 1 5 3 4 4 1 1 3 2 4 5 1

LRU

5

1

2

28

LRU with k = 3

4 1 2 1 5 3 4 4 1 1 3 2 4 5 1

LRU

5

1

2

miss

29

LRU with k = 3

4 1 2 1 5 3 4 4 1 1 3 2 4 5 1

LRU

5

1

3

30

LRU with k = 3

4 1 2 1 5 3 4 4 1 1 3 2 4 5 1

LRU

5

1

3

miss

31

LRU with k = 3

4 1 2 1 5 3 4 4 1 1 3 2 4 5 1

LRU

5

4

3

32

LRU with k = 3

4 1 2 1 5 3 4 4 1 1 3 2 4 5 1

LRU

5

4

3

hit

33

LRU with k = 3

4 1 2 1 5 3 4 4 1 1 3 2 4 5 1

LRU

5

4

3

miss

34

LRU with k = 3

4 1 2 1 5 3 4 4 1 1 3 2 4 5 1

LRU

1

4

3

so and so forth…

35

Competitive Ratio Claim

Claim: If the optimal sequence of choices for a size-h

cache causes m misses. Then, for the same sequence

of requests, LRU for a size-k cache causes

misses

Interpretation: If LRU had twice as much cache size as

an algorithm OPT that knew the future, it would have

at most twice the misses of OPT.

Note will prove the claim for

36

Proof of Competitive Ratio

Recursively break the sequence of inputs into phases.

Let t be the time when we see the (k+1)st different

request.

Phase 1: a1 … at-1

Let t` be the time we see the (k+1)st different element

starting from at

Phase 2: at … at’-1

4 1 2 1 5 3 4 4 1 1 3 2 4 5 1

37


4 1 2 1 5 3 4 4 1 1 3 2 4 5 1

k=

3

Phase 1 Phase 2 Phase 3 Phase 4

By construction, each phase has k distinct requests.

Q: At most how many misses does LRU have in each

phase?

A: k b/c even if it evicted everything in the k+1st

item, it would have at most k misses.

38


4 1 2 1 5 3 4 4 1 1 3 2 4 5 1

Phase 1 Phase 2 Phase 3 Phase 4

Q: What’s the minimum misses that any size-h

cache must have in any phase?

A: k-h b/c k distinct items will be in the cache at

different points during the phase, so at least k-h of

them must trigger misses.

Therefore the CR: k/k-h

Q.E.D.

39

Outline For Today

1. Caching


161




40

Parallel Algorithms

Question: Which problems are parallelizable,

which are inherently sequential?Parallelizable: Connected components, sorting,

selection, many computational geometry problems all

have parallel algorithms

(Believed To Be) Inherently Sequential (P-complete):

DFS

Horn-satisfiability

Conway’s Game of Life, and others.

41

2 Common Computational ModelsModel 1: Shared Memory (PRAM):

Single Machine

Memory

CPU1CPU2 CPUk…

Each Time Step, each processor:

Can read a location of memory

Can write to a location of memory

Q: How much time does it take to solve a

computational problem with polynomial #

processors?

42

Example 1: Finding the min in an array 4 9 2 3 5

Memory

CPU(1,2) CPU(1,3) CPU(4, 5)…

1 2 3 4 5

0 0 0 0 0

There are n(n-1) processors, one for each pair (i, j)

Initially allocate an array of size n all 0

Step 1: Each cpu(i, j) compares i and j

If i < j, then write 1 to j o.w. write 1 to location i

Step 2: return the item whose output memory

location is 0

43

Example 1: Finding the min in an array 4 9 2 3 5

Memory

CPU(1,2) CPU(1,3) CPU(4, 5)…

1 2 3 4 5

1 1 0 1 1

Output

A[3]=2

44

Model 2: Distributed Memory

Machine 1 Machine 2 Machine k

Input Partition1

Input Partition2

Input Partitionk

3 5 8 10 1 11 2 9 3 7 6 0

Each machine

performs local computation

send/receives messages to/from other machines

can be synchronously or asynchronously

Q: How much communication is necessary?

Q: How many synchronizations is necessary?

45

Recap: Bellman-Ford

∀ v, and for i={1, …, n}

P(v, i): shortest s v path with ≤ i edges (or ⤳

null)

L(v, i): w(P(v, i)) (and +∞ for null paths)

L(v, i) =

min

L(v, i-1)

minu: ∃(u,v)∈E : L(u, i-1) + c(u,v)

46

Example: Distributed Bellman-Ford

A C

D

FB

E

5

3

-6

2

-1

2

47


∞

0 ∞

∞

∞∞

A C

D

FB

E

5

3

-6

2

-1

2

48


∞

0 5

∞

∞∞

A C

D

FB

E

5

3

-6

2

-1

2

49


∞

0 5

8

7-1

A C

D

FB

E

5

3

-6

2

-1

2

50


1

0 5

8

7-1

A C

D

FB

E

5

3

-6

2

-1

2

51


1

0 5

0

7-1

A C

D

FB

E

5

3

-6

2

-1

2

52


1

0 5

0

7-1

A C

D

FB

E

5

3

-6

2

-1

2

Termination When No Vertex Value Changes

53

Parallel Algorithms

Parallel Computing: CS 149

A systems course.

Teaches parallel technologies but also

algorithms.

54

Outline For Today

1. Caching


161




55

Linear Programming (CS 261/361)

Optimization Problem of following

structure: maximize x1 + x2

subject to

x1

+ 2x2 ≤ 1

2x1 + x2 ≤ 1

x1

≥ 0

x2

≥ 0

56

Geometric Interpretation

Constraint 2= 2x1 + x2 ≤ 1

x1

x2

Constraint 1= x1 + 2x2 ≤ 1

Feasible Solution Set

**Opt Solution**

57

Linear Programming Applications

Tons! Lots and lots of problems can be

solved or approximated with LP! Vertex Cover

Set Cover

Load Balancing

Lots of problems in manufacturing/operations

research/finance, etc..

Covered in CS 261/361

**Invented By George Dantzig.**

58

Outline For Today

1. Caching


161




59

Approximation Techniques (including LP,

SP: Semidefinite Programming)

Many more Approximation Algorithms

Approximation Algorithms (CS 261/CS 361)

60

Commonly appearing general

randomization techniques.

Probabilistic Method

Markov Chains

**Chernoff Bounds**

Very very cool/elegant algorithms!

Randomized Algorithms (CS 365)

61

Competitive Ratio

Average-Case Analysis

Instance Optimality

Smoothed Analysis, and others…

Beyond Worst-Case (CS 369)

62

Resource Lower Bounds on Computational

Problems

Algorithms study what can be computed and

with how much resource?

Complexity studies what cannot be computed

with how much resource?

Turing

Machines/P-NP/Circuit-Complexity/Randomize

d-Complexity/Polynomial Hierarchy/Space

Complexity/PCP/One-way Functions

Complexity Theory (CS 254)

63

Power of Quantum Computers over Classic

Computers

Quantum Teleportation

Quantum Circuits

Quantum Algorithms: Fourier

Transform/Factoring/Search, etc…

Quantum Error Correction

Quantum Computing (CS 259Q)

64

Sequencing

Genome assembly

Gene sampling

Gene finding

Gene comparison

Gene regulation, etc…

Computational Genomics (CS 262)

65

Geometric modeling of physical objects

Search in high-dimensional spaces

Triangulation of a set of points

Image Processing/Graphics/Vision

Computational Geometry (CS 268)

66

Design and Analysis of Computational

Problems In Strategic Environments

Key Analysis Concept: “Price of Anarchy”

Also Studies Complexities of Finding

Equilibria

Auctions

Traffic Routing

Multi-agent Systems, and others…

Algorithmic Game Theory (CS 364)

67

Randomized Algorithms (CS 365) (Chernoff

Bounds)

Linear Programming (CS 261/361)

Complexity Theory (CS 254)

My Recommendations For Future Classes

68

Tim Roughgarden

Keith Schwarz

Billy, Mike & Yiming

Acknowledgements

69

Thank you!

Good luck in the final!

caching parallel computational models other topics in algorithms wednesday, august 13 th

Documents