lecture 12: knapsack problems€¦ · •mathematical formulation of 0–1 knapsack problem •a...
TRANSCRIPT
-
Lecture 12:Knapsack Problems
Prof. Krishna R. PattipatiDept. of Electrical and Computer Engineering
University of ConnecticutContact: [email protected]; (860) 486-2890
© K. R. Pattipati, 2001-2016
-
Outline
VUGRAPH 2
• Why solve this problem?
• Various versions of Knapsack problem
• Approximation algorithms
• Optimal algorithms
Dynamic programming
Branch-and-bound
-
• A hitch-hiker has to fill up his knapsack of size V by selecting from among various possible objects those which will give him maximum comfort
• Suppose want to invest (all or in part) a capital of V dollars among n possible investments with different expected profits and with different investment requirements to maximize total expected profit
• Other applications: cargo loading, cutting stock
• Mathematical formulation of 0–1 Knapsack problem
• A knapsack is to be filled with different objects of profit pi and of weights wiwithout exceeding the total weight V
• pi & wi are integers (if not, scale them)
• wi≤ 𝑉, ∀𝑖
• σ𝑖=1𝑛 𝑤𝑖 > 𝑉
0–1 Knapsack problem
VUGRAPH 3
1
1
max
s.t.
{0,1}
n
i i
i
n
i i
i
i
p x
w x V
x
-
• Bounded Knapsack problem
Suppose there are bi items of type i
⇒ Change 𝑥𝑖 ∈ {0, 1} to 0 ≤ 𝑥𝑖 ≤ 𝑏𝑖 and xi integer
Unbounded Knapsack problem
⇒ 𝑏𝑖 = ∞, 0 ≤ 𝑥𝑖 ≤ ∞, xi integer
Subset-sum problem
o Find a subset of weights whose sum is closest to, without exceeding capacity
o Knapsack problem with wi = pio Subset-sum is NP-hard ⇒ knapsack is NP-hard
Change-making problem
o bi finite ⇒ bounded change-making problem
o bi = ∞ ⇒ unbounded change-making problem
1
1
max
s.t.
0 , an integer, 1, ,
n
i
i
n
i i
i
i i i
x
w x V
x b x i n
Other related formulations
VUGRAPH 4
1
1
max
s.t.
{0,1}
n
i i
i
n
i i
i
i
w x
w x V
x
-
• 1-dimensional knapsack with a cost constraint
• Multi-dimensional knapsack in which profit and weight of each item depends on the knapsack selected for the item
Other related formulation
VUGRAPH 5
1
1
1
max
s.t.
{0,1}
n
i i
i
n
i i
i
n
i i
i
i
p x
w x V
c x C
x
1 1
1
1
max
s.t. ; 1, ,
1; 1, ,
{0,1}; 1, , ; 1, ,
n m
i ij
i j
n
i ij
i
m
ij
j
ij
p x
w x V j m
x i n
x i n j m
1 1
1
1
max
s.t. ; 1, ,
1; 1, ,
{0,1}; 1, , ; 1, ,
n m
ij ij
i j
n
ij ij j
i
m
ij
j
ij
p x
w x V j m
x i n
x i n j m
• Multi-dimensional knapsack (m knapsacks)
-
Other related formulation
VUGRAPH 6
• Loading problem or variable-sized bin-packing problem
Given n objects with known volumes wi & m boxes with limited capacity cj, j=1,…,m, minimize the number of boxes used
yi = 1 if box j is used
xij = 1 if object i is put in box j
cj = c ⇒ bin-packing problem
• See S. Martello and P. Toth, Knapsack Problems: Algorithms and Computer Implementation, John Wiley, 1990 for an in-depth survey of these problems
• Here we consider only 0–1 Knapsack problem
1
1
1
min
s.t. 1; 1, ,
; 1, ,
, {0,1}
n
j
j
m
ij
i
n
i ij j j
i
j ij
y
x i n
w x c y j m
y x
-
• Let us consider relaxed LP version of Knapsack problem
Gives us an upper and lower bound on the Knapsack problem
Assume that objects are ordered as
LP relaxation to find an upper bound
Dual of relaxed LP
Relaxed LP version of Knapsack problem
VUGRAPH 7
1 2
1 2
n
n
p p p
w w w
*
max relax 0 1 max
s.t. s.t.
{0,1} constraints 0 1
i i i i
i i
i i i i
i i
i i
p x p x
w x V w x V
x x
f
LPf
1
min
s.t.
, 0
n
i
i
i i i
i
V
w p
1
min
s.t.
0
n
i
i
i i i
i
V
p w
01
1* 1
1 11
* 1
1
min max(0, )
; :
max(0, )
n
i i
i
r rr
i i
i ir
ri i i
r
V p w
pr w V w
w
pp w
w
-
• Complementary slackness condition
𝑥𝑖 = 1 ⇒𝜇𝑖 > 0 and 𝑤𝑖𝜆 + 𝜇𝑖 = 𝑝𝑖 𝑥𝑖 < 1 ⇒ 𝜇𝑖 = 0
• Optimal solution is as follows
• Clearly, 𝑓𝐿𝑃∗ = optimal dual objective function value
Relaxed LP version of Knapsack problem
VUGRAPH 8
1
1 1
11 1
1
* * 1
1 1 1
if then
1, 1, , = ( )
0, 2, , = 0
; = 0
one upper bound is:
optimal
r r
i i
i i
i i i i
i i
r
i
ir r
r
r rr
LP i i
i i r
w V w
x i r p w
x i r n
V w
xw
pf f p V w
w
1
1
1
1
; 1, ,
r
r
ii i r
r
p
w
wp p i r
w
-
• LP relaxation solution provides the upper bound 𝑈1 = int 𝑓𝐿𝑃∗ ≤ 2𝑓∗
Both [p1, p2,…, pr] and pr+1 are feasible Knapsack solutions
Feasible solution ≤ 𝑓∗
𝑈1 ≤ sum of two feasible solution ≤ 2𝑓∗
• We can obtain an even tighter bound than LP relaxation (Martello and Toth)
Consider the possibility that xr+1 = 1 or xr+1 = 0
LP relaxation upper bound
VUGRAPH 11
1 2
1
2
1 2
1
1
111
1
1
0 could be a fraction
ˆ
1 could be a fraction
r r
r
iri
i r
i r
r r
r
i rri
i r r
i r
x x
V w
U p pw
x x
V w w
U p p pw
1
1
1
1
r
r rri
i r r
i r
w V w
p p pw
-
• Clearly,
• Why is this a better upper bound than LP relaxation? Clearly, 𝑈 ≤ 𝑈1 Can show that
LP relaxation upper bound
VUGRAPH 12
*2 ˆmax ,U U U f
11
1
11
0r
r ri
ir r
p pU U w V
w w
-
Knapsack Example
VUGRAPH 13
n = 8
p = [15, 100, 90, 60, 40, 15, 10, 1]
w = [2, 20, 20, 30, 40, 30, 60, 10]
V = 102
Optimal solution: x = [1, 1, 1, 1, 0, 1, 0, 0]
Optimal value: f* = 280
LP relaxation: (r + 1) = 5 ⇒ U1= 265 + (30)(40)
40= 295
Bound 𝑈 = 265 + (30)(15)
30= 280
Bound ෩𝑈 = 245+(20)(60)
30= 285
Maximum of latter two bounds = 285
• These bounds can be further improved by solving two continuous relaxed LPs with the constraints that xr+1 = 1 or xr+1 = 0, respectively
• Algorithm for solving Knapsack problem Approximation algorithms Dynamic programming Branch-and-bound
• Although the problem is NP-hard, can solve problems of size n = 100,000 in a reasonable time
• Approximate algorithms…lower bounds on the optimal solution Can obtain from a greedy heuristic
෩𝑈 =𝑉 − σ𝑖=1
𝑟−1𝑤𝑖 − 𝑤𝑟+1𝑤𝑟
𝑝𝑟
𝑈 =𝑉 − σ𝑖=1
𝑟 𝑤𝑖𝑤𝑟+2
𝑝𝑟+2
-
Greedy heuristic
VUGRAPH 14
• 0th order greedy heuristic
Load objects on the basis of 𝑝𝑖
𝑤𝑖
• Kth order greedy heuristic We can obtain a series of lower bounds by assuming that a certain
set of objects J are in the knapsack where
And assigning the rest on the basis of greedy method
This is basically rollout concept of dynamic programming
*
1:
1
&
(0)i i
i i
i
ip p
iw w
w V
L p f
i
i J
w V
-
kth order Greedy heuristic
VUGRAPH 15
The bound can be computed as follows:
o 𝑧 = 𝑉 − σ𝑖∈𝐽𝑤𝑖
o y=σ𝑖∈𝐽 𝑝𝑖
o For 𝑖 = 1,… , 𝑛 do If (𝑖 ∉ 𝐽 & 𝑤𝑖 ≤ 𝑧) then
𝐽 = 𝐽 ∪ 𝑖
𝑧 = 𝑧 − 𝑤𝑖 𝑦 = 𝑦 + 𝑝𝑖
End if
o End do
o 𝐿(𝐽) = 𝑦 ≤ 𝑓∗
-
• Idea: what if I consider all possible combination of objects of size |J| = k & find the best lower bound & use it as a solution of the Knapsack problem
If |J| = n, optimal
o We can control the complexity of the algorithm by varying k
k-approximation algorithm…Knapsack(k)
o f (k) ← 0
Enumerate all subset 𝐽 ⊂ {1,… , 𝑛} such that
Note: in practice, k = 2 would suffice to produce a solution within 2-5% of optimal
• Example
k-approximation algorithm
VUGRAPH 16
| | and do
( ) max ( ), ( )
i J
J V
f k f k L J
1 2 3 4 5 6
1 2 3 4 5 6
max 100 50 20 10 7 3
s.t. 100 50 20 10 7 3 165
1,
(0) 163 0 order heuristic
1: ({1}) 163, ({2}) 163, ({3}) 140, ({4}) 163, ({5}) 160, ({6}) 163
best 1 o
i
i
th
st
x x x x x x
x x x x x x
pi
w
L
k L L L L L L
rder heuristic solution = 163... in fact, this is optimal!
-
k-approximation algorithm
VUGRAPH 17
• Can we say anything about the performance of the approximation algorithm? Yes! The recursive approximation algorithm Knapsack(k) provides a
solution 𝑓(𝑘) ∋
Before we provide a proof, let us consider its implications
o If want a solution within 𝜖 of optimal
o Time complexity
Number of time greedy (0) is executed
Each greedy takes O(n) operations
*
*
( ) 1
1
f f k
f k
1 11
1k
k
11
0 0
1( ) ( )
1
kk ki k
i i
n nn O n O n
i n
-
• Let R* be the set of objects included in the knapsack in an optimal solution
• If | R*|≤ 𝑘, our approximation algorithm would have found it…so, assume otherwise
• Suppose ( Ƹ𝑝𝑖 , ෝ𝑤𝑖), 1 ≤ 𝑖 ≤ |𝑅∗| be the set of objects in R*
• Assume that we order these objects as follows:
First 𝑘: Ƹ𝑝1 > Ƹ𝑝2 > ⋯ > Ƹ𝑝𝑘 are the largest profits
The rest: ො𝑝𝑖ෝ𝑤𝑖>
ො𝑝𝑖+1ෝ𝑤𝑖+1
, 𝑘 < 𝑖 < |𝑅∗|
•
•
Proof of the bound: setting the stage
VUGRAPH 18
* *
* &i ii R i R
p f w V
*R k
*
*
1 2 | |
* *
1
1
*
ˆ ˆ ˆ
ˆ ˆ ˆ( 1) ; 1,...,| |
ˆ1
R
k
i k k t
i
k t
p p p f
f p p k p t k R
fp
k
-
Proof of the k-approximation bound - 1
VUGRAPH 19
• Let us look at what k-approximation algorithm does … it must have looked at the set J of largest prices in R* at least once … let
• Let us consider what happens when k-approximation algorithm executed this iteration
Suppose that the approximation algorithm did not include an object that is in the optimal solution
Let l be the first such object, that is, 𝑙 ∉ 𝐽
Suppose l corresponds to ( Ƹ𝑝𝑚, ෝ𝑤𝑚) in R*
• Why was l not included in the Knapsack(k)?
Residual capacity of knapsack 𝑧 < 𝑤𝑙 = ෝ𝑤𝑚 Must have included (m - 1) tasks
1
ˆk
J i i
i J i
f p p
1 1
1 1 1
1
ˆˆ ˆ ˆ( )
ˆ
ˆ( )
J
J
S
k m mm
i i i
i i k im
f
k
i
i
f
pf k p p V w z
w
f k p S
-
VUGRAPH 20
• Also, from LP relaxation bound
• By definition
*
1 1
1 1
ˆˆ ˆ ˆ
ˆ
R m mm
i i i
i m i im
pp p V w
w
*
*
*
1
1
1
1
1
*
* *
ˆ ˆ
ˆ ˆˆ
ˆ ˆ
ˆ
ˆ
ˆ ˆ( )
ˆ( ) 1
1
R
J i
i k
Rm
J i i
i k i m
mm m
J i
im m
mJ
m
J m m
m
f f p
f p p
p pf S V w
w w
pf S z
w
f S p f k p
pf f k
f f k
Proof of the k-approximation bound - 2
-
VUGRAPH 21
• Since Ƹ𝑝𝑚 is one of Ƹ𝑝𝑘+1⋯ Ƹ𝑝 𝑅∗ ⇒ Ƹ𝑝𝑚 ≤ ҧ𝑝 = 𝑘 + 1st largest element of Ƹ𝑝1⋯ Ƹ𝑝𝑚
• Example
p = [11, 21, 31, 33, 43, 53, 55, 65] optimal: 1 2 3 5 6
w = [1, 11, 21, 23, 33, 43, 45, 55] f* = 159, σ𝑤𝑖 = 109 < 110
V = 110
k = 0 ⇒ x = [1, 1, 1, 1, 1, 0, 0, 0], f = 139 ⇒𝑓∗−𝑓
𝑓∗=
20
159= 0.126, σ𝑤𝑖 = 89
k = 1 ⇒ x = [1, 1, 1, 1, 0, 0, 1, 0], f = 151 ⇒𝑓∗−𝑓
𝑓∗=
8
159= 0.05, σ𝑤𝑖 =101
k = 2 ⇒ x = [1, 1, 1, 0, 1, 1, 0, 0], f = 159 ⇒𝑓∗−𝑓
𝑓∗=
0
159= 0, σ𝑤𝑖 =109
*
* *
*
*
ˆ ˆ( )
( ) ( )
( ) 1min ,
1 ( )
m mf f k p p p
f f f k f k
f f k p
f k f k
Proof of the k-approximation bound - 3
-
Example
VUGRAPH 22
Usually k = 1 or k = 2 works OK (within 2-5% of optimal)
1 2 3 4
1 2 3 4
max 9 5 3
s.t. 7 4 3 2 10
9 5 3 1note: (OK)
7 4 3 2
0 : ({0}) 12, [1,0,1,0]
1: ({1}) 12, ({2}) 9, ({3}) 12, ({4}) 10
2 : ({1,3}) 12, ({1,4}) 10, ({2,3}) 9, ({2,4}) 10, ({3,4}) 10,etc.
x x x x
x x x x
k L x
k L L L L
k L L L L L
f
*(2) 12 f
-
Dynamic programming approach
VUGRAPH 23
• Views the problem as a sequence of decisions related to variables x1, x2,…, xn
• That is, we decide whether x1 = 0 or 1, then consider x2, and so on
• Consider the “generalized” Knapsack problem:
• Actually want to solve: Knap (1, n, V)
• To solve Knap (1, n, V), the DP algorithm employs the principle of optimality
“The optimal sequence of decisions has the property that, whatever the initial state and past decisions are, the remaining decisions must constitute an optimal decision sequence with regard to the state resulting from the current decision”
max
s.t. Knap( , , )
{0,1}, , ,
l
i i
i k
l
i i
i k
i
p x
w x U k l U
x i k l
-
• Suppose 𝑓0∗(𝑉) = optimal value for Knap (1, n, V)
• 𝑓𝑗∗(𝑈) = optimal value for Knap (j + 1, n, V), 1 ≤ 𝑗 ≤ 𝑛 − 1
• Clearly, 𝑓𝑛∗ 𝑈 = 0, ∀𝑈
• Let us look at 𝑓0∗(𝑉) … suppose we make the decision for xi
• If 𝑥1 = 0, 𝑥2⋯𝑥𝑛 must be the optimal solution for Knap (2, n, V) = 𝑓1∗(𝑉)
• If 𝑥1 = 1, 𝑥2⋯𝑥𝑛 must be the optimal solution for Knap (2, n, V – w1) = 𝑓1∗(𝑉 − 𝑤1)
• Similarly, 𝑓𝑗∗(𝑈) = optimal solution of Knap (j + 1, n, U)
Backward DP approach
VUGRAPH 24
* * *0 1 1 1 1
1 1
( ) max ( ), ( )
0 1
f V f V f V w p
x x
* * *1 1 1 1
1 1
( ) max ( ), ( ) , 1, ,0
0 1
( ) 0,
j j j j j
j j
n
f U f U f U w p j n
x x
f U U
-
Forward DP
VUGRAPH 25
• Start with 𝑓𝑛∗ 𝑈 = 0, ∀𝑈 = 0, 1, 2, … , 𝑉…successively evaluate
𝑓𝑛−1∗ 𝑈 , ∀𝑈, then 𝑓𝑛−2
∗ 𝑈 ,∀𝑈, etc.
• Note: decision xj+1 depends on xj+2,…, xn ⇒ backward recursion
• Alternately, suppose we know the optimal solution to Knap (1, j, U) … we could have solved it in one of two ways:
xj = 0 and knew the solution to Knap (1, j–1, U) = 𝑆𝑗−1∗ 𝑈
xj = 1 and knew the solution to Knap (1, j–1, U–wj) = 𝑆𝑗−1∗ 𝑈 − 𝑤𝑗 + 𝑝𝑗
So we can get a forward recursion
where
Note: decision xj depends on x1,…, xj-1 ⇒ forward recursion
* * *1 1( ) max ( ), ( )j j j j jS U S U S U w p
0 0( ) 0, 0 and ( ) , 0S U U S U U
-
Example
VUGRAPH 26
• p = [1, 2, 5]; w = [2, 3, 4]; V = 6
• Computational load and storage O(nV)
• It is considered a pseudo-polynomial algorithm, since V is not bounded by a log2V function
• By encoding x as a bit string, can reduce storage to 𝑂 1 +𝑛
𝑑𝑉 , where d is
the word length on the computer (see Martello and Toth’s book)
U 0
(S0, x0)
1
(S1, x1)
2
(S2, x2)
3
(S3, x3)
0 (0, –) (0 ,0) (0 ,0) (0, 0)
1 (0, –) (0 ,0 ) (0, 0) (0, 0)
2 (0 ,–) (1 ,1) (1, 0) (1, 0)
3 (0, –) (1, 1) (2 ,1) (2, 0)
4 (0 ,–) (1, 1) (2, 1) (5, 1)
5 (0, –) (1, 1) (3, 1) (5, 1)
6 (0, –) (1, 1) (3, 1) (6 ,1)
* * *1 1( ) max ( ), ( )j j j j jS U S U S U w p
0 0( ) 0, 0 and ( ) , 0S U U S U U
1 2 3
*
1; 0; 1
6
x x x
f
-
• Finds the solution via a systematic search of the solution space
• For Knapsack problem, the solution space consists of 2n vectors of 0s and 1s
• The solution space can be represented as a tree
Leaves correspond to “potential solutions,” not necessarily feasible
Key:
o Don’t want to search the entire tree
o Don’t want to generate infeasible solutions
o Don’t want to generate tree nodes that do not lead to a better solution than the one on hand (⇒ a bounding function)
Branch-and-bound method: Basic Idea
VUGRAPH 27
x1
x2
x3
x4
1
1
1
1
1 1
1
1
1
1
1
1 1 0
0
0
0
0
00 1
0
0
00
0
0 0 1 0
-
• Bounding function can be derived from the LP relaxation Given the current contents of the knapsack J with profit
P and weight W, we can construct the bounding function as follows:o Suppose k is the last object considered, i.e., p & W correspond
to 𝑦1⋯𝑦𝑘 (𝑦𝑖 = temp 𝑥𝑖), then
o If the UB ≤ current best solution, then further search from a tree node is worthless
So, backtrack ⇒ move to the right, if it was in the left one or go back to the first variable with value
B&B for knapsack problem
VUGRAPH 28
1
1
max
s.t.
0 1
n
i i
i k
n
i i
i k
i
p p y
w y V W
y
-
• We will explain the algorithm by means of an example (Syslo, Deo, and Kowalik)
• Initially p = W = 0, k = 0 & f = –1 Partial solution: y1 = 1, y2 = 1 with profit = 5 and
Weight = 3 ⇒ p = 5, W = 3, k + 1 = 3
Bound: b = 5 + int(2)(6)
5= 7
Bound > f … perform a forward move
Example
VUGRAPH 29
1 2 3 4
1 2 3 4
max 2 3 6 3
s.t. 2 5 4 5
{0,1}i
x x x x
x x x x
x
-
• y3 = 0 (since can not fit in it) and y4 = 1 is infeasible
• Set y4 = 0 and k + 1 = 5 > n = 4 and f = –1, the current solution is the best found so far … so, set f = 5 and k = 4
• Backtrack to the last object assigned to knapsack … remove the object … y2 = 0 ⇒ p = 2, W = 1, y3 = 1 fails ⇒ y3 = 0 … but, y4 = 1 O.K. ⇒ y = [1, 0, 0, 1] and p = 5 … this does not improve the current best solution of f = 5
• Backtrack to y1 … set y1 = 0 after assigning y2 = 1, it returns bound b = 6 … b > f, we do another forward move … the next call to bound results in a bound 𝑏 ≤ 𝑓 ⇒ backward move
• Backtrack to y2 … set y2 = 0 after assigning y3 = 1, it returns bound b = 6 and a solution p = 6, W = 5, which is better than the best solution f…. Actually you could you have stopped here because upper bound = feasible solution
• Finally, y3 = 0, and bound assigns y4 = 1 and returns 3 … since there are no other objects in the knapsack to be removed, the algorithm terminates
• There exist variations that provide better computational performance … see the book by Martello and Toth
Example
VUGRAPH 30
y1
y2
y3
y4
1
1
7
6
6
1
1
1
0
0
0
0
0
0
00
0
6
1
5 5
6
5 6 3
0
f = 5 5 ≤ 𝑓 5 ≤ 𝑓 f = 6 3 < f
-
Summary
VUGRAPH 31
• Various versions of Knapsack problem
• Approximation algorithms
• Optimal algorithms
Dynamic programming
Branch-and-bound