mit and james orlin © 2003 1 dynamic programming 2 –review –more examples

MIT and James Orlin © 2003

1

Dynamic Programming 2

– Review–More examples


2

Match game example

Suppose that there are 20 matches on a table, and the person who picks up the last match wins. At each alternating turn, my opponent or I can pick up 1, 2 or 6 matches. Assuming that I go first, how can I be sure of winning the game?


3

Determining the strategy using DP n = number of matches left (n is the state/stage) g(n) = 1 if you can force a win at n matches.

g(n) = 0 otherwise g(n) = optimal value function.

At each state/stage you can make one of three decisions: take 1, 2 or 6 matches.

g(1) = g(2) = g(6) = 1 (boundary conditions) g(3) = 0; g(4) = g(5) = 1.

The recursion: g(n) = 1 if g(n-1) = 0 or g(n-2) = 0 or g(n-6) = 0;

g(n) = 0 otherwise. Equivalently, g(n) = 1 – min (g(n-1), g(n-2), g(n-6)).


4

1 2 3 4 5 6 7 8 9 10

11 12 13 14 15 16 17 18 19 20

1 2 3 4 5 6 7 8 9 10

11 12 13 14 15 16 17 18 19 20

Winning and Losing Positions


5

Principle of Optimality

Any optimal policy has the property that whatever the current state and decision, the remaining decisions must constitute an optimal policy with regard to the state resulting from the current decision.

Whatever node j is selected, the remaining path from j to the end is the shortest path starting at j.


6

Capital Budgeting, again

Investment 1 2 3 4 5 6

Cash Required (1000s)

$5

$7

$4

$3

$4

$6

NPV added (1000s)

$16

$22

$12

$8

$11

$19

Investment budget = $14,000


7

Stages, and the DP recursion

Solve the knapsack problem as a sequence of problems

At stage k we consider whether to add item k or not to the knapsack.

Let f(k,B) be the best NPV limited to stocks 1, 2, …, k only and using a budget of at most B.

Knapsack problem via DP


8

Solving the Stockco Problem

At stage 1 compute for each budget from 0 to $14,000 compute the best NPV while limited to stock 1.

At stage 2 compute for each budget from 0 to $14,000 compute the best NPV while limited to stocks 1 and 2.

At stage 3 compute for each budget from 0 to $14,000 compute the best NPV while limited to stocks 1, 2 and 3.


9

Capital budgeting in general

Max j=1..n cj xj

s.t j=1..n aj xj b

x binary

Let f(k, B) = Max j=1..k cj xj

s.t j=1..k aj xj B

x binary


10

The recursion

f(0,0) = 0; f(0,k) is undefined for k > 0

f(k, B) = min ( f(k-1, B), f(k-1, B-ak) + ck) either item k is included, or it is not

The optimum solution to the original problem is max { f(n, B) : 0 B b }.

Note: we solve the capital budgeting problem for all right hand sides less than b.


11

The Knapsack ProblemIs there a feasible solution to

31 x + 35 y + 37 z = 422

x, y, z are non-negative integers.

let f(1, j) = T if there is a solution to 31 x = j, f(1, j) = F if there is no solution to 31 x = j. where x is a non-negative integer

let f(2, j) = T if there is a solution to

31 x + 35 y = j, x and y are non-negative integer


12

The Knapsack ProblemIs there a feasible solution to

31 x + 35 y + 37 z = 422


let f(3 ,j) = T if there is a solution to

31 x + 35 y + 37 z = j, x, y, and z are non-negative integer

Note: there are three stages to this problem.


13

The recursion for the knapsack problem

Is there a feasible solution to

31 x = 422

x non-negative integer?

f(0,0) = T, f(0,k) = F for k > 0

f(1,k) = T if f(0,k) = T or f(1, k-31) = T for k = 31 to 422

f(1,k) = F otherwise.

Does this really work?


14



31 x + 35 y = 422

x, y are non-negative integers.

f(2,k) = T if f(1,k) = T or f(2, k-35) = T for k = 0 to 422

f(2,k) = F otherwise


15



31 x + 35 y + 37 z = 422

x, y, z are non-negative integers?

f(3,k) = T if f(2,k) = T or f(3, k-37) = T

f(3,k) = F otherwise


16

The DP again, without stagesIs there a feasible solution to

31 x + 35 y + 37 z = 422


let g(j) = T if there is a solution to

31 x + 35 y + 37 z = j, x, y, and z are non-negative integer

Then g(0) = T.

For j > 0, g(j) is true if i. g(j-31) is true orii. g(j-35) is true oriii. g(j-37) is true


17

Optimal Capacity Expansion: What is the least cost way of building plants?

Year Cum. Demand Cost per plant in $millions

2002 1 54

2003 2 56

2004 4 58

2005 6 57

2006 7 55

2007 8 52

Cost of $15 million in any year in which a plant is built. At most 3 plants a year can be built


18

Cost of Building plantscost of building in each year

3 177 183 189 186 180 1712 123 127 131 129 125 1191 69 71 73 72 70 670 0 0 0 0 0 0

years to go 6 5 4 3 2 1 0

Minimum number of plants needed

# of plants 1 2 4 6 7 8years to go 6 5 4 3 2 1 0


19

Step 1. For each year, identify those table entries that are feasible, taking into account the lower bounds and the fact that at most 3 plants can be built per year.

876543210

6 5 4 3 2 1 0Years to go

(Y, k) is a state denoting having k plants with Y years to go.


20

876543210

6 5 4 3 2 1 0Years to go

Step 2. f(Y, k) = remaining cost of capacity expansion starting with k plants with Y years to go.

Fill in the entries for 0 years to go. Then fill in the entries for 1 year to go. Then fill in entries for 2 years to go, and continue.


21

Determining state and stage

Stage: Y = years to go State: number of plants f(j, Y) = optimum cost of capacity

expansion starting with j plants with Y years to go

We want to compute f(0, 6).

What is f(j, 0) for different j?


22

The recursion Let g(Y, k) be the cost of building k plants with Y

years to go. Let l(Y) be the minimum number of plants needed

with Y years to go. Let M(Y) be the maximum number of plants that can

be built with Y years to go.

Let f(Y,k) be the remaining cost of capacity expansion starting with k plants with Y years to go.

What is f(0,k)? Can you compute f(Y,k) assuming that f(Y-1, j) is already computed for each j?


23

Summary for Dynamic Programming

Recursion Principle of optimality states and stages and decisions useful in a wide range of situations– games – shortest paths– capacity expansion–more will be presented in next lecture.


24

Dynamic Programming Review

Next lecture: DP under uncertainty.

Applications –meeting a financial goal

(a very simplified version)– getting a plane through enemy territory

mit and james orlin © 2003 1 dynamic programming 2 –review –more examples

Documents