Linear Programming and Simplex Algorithm

Reference:

Numerical Recipe Sec. 10.8

2

Content

• Linear programming problems• Simplex algorithm• QSOpt solver• LP Applications

– Collision detection– Melodic similarity– Underdetermined linear systems

3

The Diet Problem

• Dietician preparing a diet consisting of two foods, A and B.

Cost Protein Fat Carbohydrate

A 0.60 20 12 30

B 0.40 30 6 15

Minimum requirement: protein 60g; fat 24g; carbohydrate 30gLooking for minimum cost diet

4

Linear Programming

0,

301530

246 12

603020 ..

40.060.0min

21

21

21

21

21

xx

where

xx

xx

xxts

xxC

0)(

0)(

)(min

xg

xh

xfsubject to

Cp. NLP

5

The ProblemMaximize

Subject to

N: dimension of xM: number of constraints (M = m1+m2+m3)

N: dimension of xM: number of constraints (M = m1+m2+m3)

6

Theory of Linear Optimization

Terminology: Feasible vector Feasible basic vector Optimal feasible vector

7

Theory (cont)

• Feasible region is convex and bounded by hyperplanes

• Feasible vectors that satisfy N of the original constraints as equalities are termed feasible basic vectors.

• Optimal occur at boundary (gradient vector of objective function is always nonzero)

• Combinatorial problem: determining which N constraints (out of the N+M constraints) would be satisfied by the optimal feasible vector

8

Simplex Algorithm (Dantzig 1948)

• A series of combinations is tried to increase the objective function

• Number of iterations less than O(max(M,N))• Worst case complexity is exponential (in number

of variables); yet has polynomial smoothed complexity (and works well in practice)

9

The Canonical ProblemMaximize

Subject to

N: dimension of xM: number of constraints (M = m1+m2+m3)

N: dimension of xM: number of constraints (M = m1+m2+m3)

I.II.

III.

11

QSopt [ref]

• Contains a GUI and a callable library

12

LP Format Example

0,

301530

246 12

603020 ..

40.060.0min

21

21

21

21

21

xx

where

xx

xx

xxts

xxC Problem smallExampleMinimize obj: 0.60x1 + 0.40x2Subject to c1: 12x1 + 6x2 >= 24 30x1 + 15x2 >= 30

Bounds 0 <= x1 0 <= x2End

Somehow x1 >= 0x2 >= 0

Doesn’t work!? BNF for QSOpt scripts

QSOpt Examples

13

14

Duality

• Primal problem • Dual problem

15

16

Duality

• If a linear program has a finite optimal solution then so does the dual. Furthermore, the optimal values of the two programs are the same.

• If either problem has an unbounded optimal solution, then the other problem has no feasible solutions.

17

Diet Problem

0,

301530

246 12

603020 ..

40.060.0min

21

21

21

21

21

xx

where

xx

xx

xxts

xxC

0,,,,

301530

246 12

603020 ..

40.060.0max

32121

321

221

121

21

sssxx

where

sxx

sxx

sxxts

xxC

Applications

19

The Problem

Under-determined system (infinite number of solutions)Find the solution with minimum magnitude (i.e., closest to origin)

x

bAx

min

20

Example

Approach 1: find complete solution

npc

pn

xxx

xcx

0

0

2

,

1

1

1

0110

2011

(-2,0,0) subtract its projection along (1,-1,1)

32

32

34

32

32

32

0

0

2

0

0

2

111

111

111

3

1

0

0

2

111

111

111

3

1,

1

1

1

3

1 Tqqq

0

2

32

21

xx

xx

924d(1,-1,1)

(-2,0,0)

21

Linear Programming Solution

0

wheremin,

ix

x

bAx

a

• Most simplex algorithm assume nonnegative variables

• Set

Rxzy

zyx

iii

iii

0 and 0 where

22

Example

0,0,0

3,2,1,

3,2,1,

0

2

:subject to

3min

3322

2211

ii

ii

ii

zy

iyz

iyz

yzyz

yzyz

iii yzx 0

2

0

0

0

0

0

0

3min

3322

2211

33

22

11

33

22

11

zyzy

zyzy

zy

zy

zy

zy

zy

zy

0

2

32

21

xx

xx

23

Solved by QSopt

The real minimum (-4/3, -2/3, 2/3)The LP constraint is not exactly ||x||2(but ||x||)

3d

This problem can also be solved by SVD (singular value decomposition)

24

LP in CD (narrow phase, ref)

• A pair of convex objects: each facet represented by the plane inequality aix+biy+ciz di

• If two sets of inequality (from two convex objects) define a feasible region, then a collision is detected

• The type of optimization (min|max) and the objective function used is irrelevant.

25

LP for CD

Convert to dual problem for better efficiency(?!)

Transportation Distance as a Measure to Melodic Similarity

Ref: Typke et al. 2003

27

Example Partially matched

28

Earth Mover Distance

ijjiij

j

jjjjj

n

i

iiiii

m

dyxf

uU

RuRyuyb

bbB

wW

RwRxwxa

aaA

distance over the to from weight of flow:

}0{,,,

setpoint weighted,,

}0{,,,

setpoint weighted,,

2

1

2

1

29

EMD as Linear Program

Solve by simplex

algorithm

30

EMD for Melodic Similarity

• Ground distance: Euclidean distance

• Scale time coordinate (so that they are comparable)

• Transposition: transpose one of the melodies so that the weighted average pitch is equal (not optimum but acceptable)

22jijiij ppttd

31

HW: Verify planar CD using LP

Warning: Non-negative assumption

(for some solvers)!!

Warning: Non-negative assumption

(for some solvers)!!

32

Verify planar CD using LP.

1

2

3

4

5

6 11[I] 11 .3

3[II] 3 .2

3[II] 3 .1

3

2

1

syxyx

sxx

syy

0II][0 .6

10I][ 10 .5

5II][ 5 .4

5

5

4

syxyx

sxx

syy

33

LP Solvers

• Lp_solve: MILP

• QSopt

• GLPK

35

QSopt [ref]

• Contains a GUI and a callable library

36

LP Format Example

0,

301530

246 12

603020 ..

40.060.0min

21

21

21

21

21

xx

where

xx

xx

xxts

xxC Problem smallExampleMinimize obj: 0.60x1 + 0.40x2Subject to c1: 12x1 + 6x2 >= 24 30x1 + 15x2 >= 30

Bounds 0 <= x1 0 <= x2End

[weird] Somehow x1 >= 0x2 >= 0

Doesn’t work!?

37

LP format BNF definition1

38

LP format BNF definition2

QSOpt API

40

Scenario: Read and Solve

41

Accessing a Solution

• value: objective function

• x: solution vector (columns)

• pi: dual variables• slack: slack variables• rc: reduced cost

• In our applications, we don’t really care about pi, slack, and rc.

• Set them to NULL

42

01.10.5

4.13.21.3

cmat

# of nonzeros in jth columnLocation of start entry

0 0 0

1 1

cmatind

QS_MAXor

QS_MIN

43

Modify an LP Problem

• QSnew_row• QSadd_rows• QSnew_col• QSadd_cols• QSdelete_row• QSdelete_col• QSchange_coef• QSchange_objcoef• QSchange_rhscoef• QSchange_sense

• See reference manuals for more details (Ch.5)

• Use QSwrite_prob to verify the change