control for stochastic models via diffusion approximations amy ward, ans lecture series 2008...

Control for Stochastic Modelsvia Diffusion Approximations

Amy Ward, ANS Lecture Series 2008

1.Control of High Volume Assemble-to-Order Systems• Queue-length control via tracking policies

2.Control of a Many-Server Queueing Model• Workload division

Fair Dynamic Routing in Large-Scale Heterogeneous-Server

Systems

Amy Ward

Joint work with Mor Armony

ANS Lecture II 2008

The Inverted-V Model

NK

K

K 21

• Calls arrive at rate (Poisson process).

• K server pools.

• Service times in pool k are exponential with rate k

N1

1

¹ 2 > ¹ 1

¹ > ¹

Experienced employees on averageprocess requests faster than new hires.Gans and Shen (2007)

…

The ProblemRouting: When an incoming call arrives to an empty queue, which agent pool should take the call? The objective is to minimize steady-state wait time.

¹ 2 > ¹ 1

¹ > ¹

x = y

NK

K

K 21

N1

1

…

No. The Slow-server Problem.2 servers.It is sometimes necessary to keepcustomers waiting even when the slower server is idle in order not to starve the faster server.Threshold control is optimal.Lin and Kumar (1984).

Is an exact analysis possible?

General multi-heterogeneous server case is still open.(Vericourt and Zhou, 2006)

The ProblemRouting: When an incoming call arrives to an empty queue, which agent pool should take the call?

¹ 2 > ¹ 1

¹ > ¹

x = y

Armony (2005) shows routing to the fastest server first (FSF) asymptotically minimizes the steady-state wait time.BUT … asymptotically only the slow servers have any idle time.Is this fair? Do we care?

NK

K

K 21

N1

1

…

Fairness

¹ 2 > ¹ 1

¹ > ¹

x = y

Call centers care!

Employee burnout and turnover.

Increased employee turnover leads to worse performance (Whitt 2006).

Call Centers address fairness byrouting to the server that has idled the longest (LISF).

How does LISF perform?

Do any other fair policies perform better?

NK

K

K 21

N1

1

…

The Fairness Problem

¹ 2 > ¹ 1

¹ > ¹

x = y

Minimize E[Waiting Time]

Subject to:

E[# of idle servers of pool k]= fk

E[Total # of idle servers]

*All in steady-state

NK

K

K 21

N1

1

…

How to determine f?

)1()( then ,)(/)( If

1211

221111 ff

NNfEIfEIEI kk

21

1121 implies

/)(1

NN

Nf

NEI kkk

¹ 2 > ¹ 1

¹ > ¹

x = y

2 classes.Expected waiting time is decreasing in the pool 1 idleness proportion f1. So we would like to choose high f1.

Should we ensure all servers have the same utilization?Expected utilizationof a pool k server

Any fairness criterion that involves individual serverutilization translates into a choice for f1

The same effective processing rate for all servers: 2211


¹ 2 > ¹ 1

¹ > ¹

x = y


Subject to:

E[# of idle servers of pool k] = fk


Solution Approach:

1. Solve approximating diffusion control problem.2. Translate solution to original system.

NK

K

K 21

N1

1

…

Literature Review• Conventional Heavy Traffic, Parallel Server

Systems– Harrison (1998), Bell and Williams (2001) (2005)

• The Limit Regime– Halfin and Whitt (1981)

• The Inverted V Model– Armony (2005), Tezcan (2006), Atar (2007) Gurvich and Whitt (2007)

• Fairness literature in EE and CS– Deals with fairness towards flows/customers– Avi-Itzhak et al (2006), Weirman (2007)

• Fairness literature in human resources– Deals with the effect fairness on employee

performancs– Cohen-Charash et al (2001), Colquitt et al (2001)

The Asymptotic Regime

¹ 2 > ¹ 1

¹ > ¹

x = y

0,0 ,1

, As

1

1

k

K

k k

kkk

K

k kk

aa

oaN

oN

X̂ ¸ = X ¸ ¡ N ¸p

N ¸hX̂ ¸

i += scaled queue length

hX̂ ¸

i ¡= scaled # of idle servers

(under the assumption of work conservation)

NK

K

K 21

N1

1

…

Fairness

¹ 2 > ¹ 1

¹ > ¹

x = y

Call centers care!






NK

K

K 21

N1

1

…

The Longest-Weighted-Idle Server First (LWISF) Policy

K

m mm

kk

N

N

1

• LISF might not obtain the desired idleness constraint

• fk(LISF)=

• To fix this, we propose LWISF: LWISF routes to pool k if wkik> wmim, where ik is the idle time of the server that has been idle the longest in pool k

fk=

• Proposition: LWISF asymptotically satisfies the idleness constraint

BUT: Does it minimize E[Waiting time]? Can we do better?

K

m mmm

kkk

wN

wN

1/

/

The Asymptotic Behavior of LWISF

0 ,

0 ,)( 1

x

xfxxm

K

k kk

LWISF is asymptotically equivalent to a preemptive policythat, at all times, balances the workload between server poolsby requiring the fraction of idle servers in pool k is fk,where f1+f2+ +fK=1.

For the preemptive policy . as ~ XX

The proposition proof is due to Stone’s theorem.The asymptotic equivalence is due to Atar (2007).

X has infinitesimal mean

X has infinitesimal variance 2.

The Asymptotic Equivalence

The diffusion limit is identical. There is state-space collapse.

.at poolin servers idle ofnumber theesapproximat )(

. at time servers idle ofnumber theesapproximat )(

at time waitingcallers ofnumber theesapproximat )(

tktXNf

ttXN

t.tXN

k

Note that under the preemptive policy, there is state-space collapse for each . The state is the number in system.

The LWISF policy maintains fixed ratios betweenthe number of idle servers in each pool.

Fairness

¹ 2 > ¹ 1

¹ > ¹

x = y

Call centers care!






NK

K

K 21

N1

1

…


¹ 2 > ¹ 1

¹ > ¹

x = y


Subject to:

E[# of idle servers of pool k]= fk


*All in steady-state

NK

K

K 21

N1

1

…

The Diffusion Control Problem

1

),(

)(2)(),(ˆ)0(ˆ)(ˆ

1

1

0

K

k k

K

k kk

t

u

xuuxm

tBdssusXmXtX

x = y2

},...,1{ ,)(ˆ)(ˆ)( subject to

)(ˆmin

KkXEfXuE

XE

kk

Can you guess the solution?

Threshold Control

)(2)(),(ˆ)0(ˆ)(ˆ

0if }{1

0 if ),(

thatso

})(ˆ{1)(

has ) and0(with

levelsat control dA threshol

0

1 1

1

0

11

tBdssusXmXtX

xLxLx

xuxm

LtXLtu

L L

,...,LL

t

K

k kkk

kkk

K

K

x = y2

Solving the Diffusion Control Problem

K

k kk fddXE

1

for )(ˆ

},...,1{for )(ˆ)( subject to

)(ˆ minimize

KkdfXuE

XE

kk

X̂

Step 1: Observe that

Step 2: Hence an equivalent DCP is

Step 3: We can now formulate the Lagrangian and solve.

K

k kkk dfXuEXE1

)(ˆ)()(ˆmin

Solving the Diffusion Control Problem Cont.

K

k kkk dfXuEXE1

)(ˆ)()(ˆmin

What are the correct penalty parameters?

},...,1{for )(ˆ1)(ˆ

ly,equivalent or,

,)(ˆ)(

1 KkdfLXLXE

dfXuE

kkk

kk

The ones under which

.)(ˆ)(ˆ

, control admissiblean under

)(2)(),(ˆ)0(ˆ)(ˆ

satisfies and

},,...,1{for )(ˆ)(

has that ˆany for Then,

}.,...,1{for )(ˆ1)(ˆ

such that be let and , control

thresholdunder thediffusion thebe ˆLet

:

*

0

1

*

XEXE

u

tBdssusXmXtX

KkdfXuE

X

KkdfLXLXE

LL

X

t

kk

kkk

Theorem

Browne and Whitt (1995).

How do we find the threshold levels?

Policy Translation: (2 classes)

Lx

NxL

xN

Use FSF.

No servers idle.

Number in system is x.

Use SSF.

Threshold level: L ¸ = N ¸ ¡ Lp

N ¸

xN

NxL

LxL

Lx

K

21

10

.constant positive somefor level Threshold kkk LNLNL

Use FSF excluding pool K (the fastest).

Use FSF excluding pool K-1.

Use FSF.

No servers idle.

. Assume 21 KLLL


Policy Translation: (K classes)

Asymptotic Optimality (a.o.)kK

k k

k fEI

EI

1);(

);(lim

Asymptotic Feasibility:

Asymptotic Optimality: 1. ¼ is asymptotically feasible, and2. If ¼’ is asymptotically feasible then

Conjecture 1: The preemptive Threshold Policy is a.o.(ext. of Stone’s theorem; Atar, Budhjiraja and Ramanan (2007))

But what about the non-preemptive threshold policy?Showing a.o. would require a non-continuous form of s.s.c..

);(ˆinflim);(ˆsuplim

WEWE

²-Asymptotic Optimality (²-a.o.) ²-Asymptotic Optimality:

1. ¼ is asymptotically feasible, and2. If ¼’ is asymptotically feasible

then

The ²–Threshold Policy

);(ˆinflim);(ˆsuplim WEWE

X

Death rate

slope ¹2

slope ¹1

L N

²-Threshold Policy: Construction

)(ˆ)(ˆ)(ˆ * ul XXX

)(,ˆ

X

)(ˆ)(ˆlim *

)(,0

XEXE

)(ˆ uX)(ˆ lX

)(ˆ)(ˆ *

)(, XEXE

1. Construct upper and lower bound diffusion processes.

2. Construct a process whose drift is a convex combination of and

3. Choose such that

4. The diffusion is -optimal because

)(ˆ)(ˆ *

)(, XEXE

5. Hence for any >0, there exists () such that

Policy Translation: -Threshold Policy

Lx

LxL

NxL

xN

Use FSF.

No servers idle.


Use SSF.

Threshold level: L ¸ = N ¸ ¡ Lp

N ¸

Depends

Adjustment: NN

Thm: The ²-Threshold Policy is ²-a.o.

);(ˆ);(ˆ

;ˆ and );(ˆ);(ˆ

1

21

1

fTHIETHIE

THIETHXETHQE

)(ˆ);(ˆinflim *XEQE

as ;ˆ;ˆ THXTHX

Proof:

)(ˆ);(ˆ and )(ˆ);(ˆ ** XETHXEXETHXE

1. Diffusion construction

2. Weak convergence (Stone)

3. State-space collapse (G&W) and Tightness and UI

4. Asymptotic lower bound (DCP solution)

5. Little’s Law

Asymptotic Performance (Predicted)

)(XE

1 = 1, 2 = 2, = 1, = 1.5, 2 = 2 = 3

f(L)0.000

0.100

0.200

0.300

0.400

0.500

0.0000 0.2000 0.4000 0.6000 0.8000 1.0000

Slow Server Idleness Proportion

Threshold

LWISF

FSF

f1

E [X̂ (1 )]+

Asymptotic Performance (Predicted)

0.00%

1.00%

2.00%

3.00%

4.00%

5.00%

6.00%

7.00%

8.00%

0.0 0.2 0.4 0.6 0.8 1.0

1 = 1, 2 = 2, = 1, = 1.5, 2 = 2 = 3

)(ˆ)(ˆ)(ˆ

LWISFTPLWISF XEXEXE

f(L)

Asymptotic Performance (Simulation)

1 = 1, 2 = 2, = 1, = 1.5, 2 = 2 = 3, N1=300, N2=200, ¸=674

A Simulation Comparison of the Threshold and LWISF Policy

0.0000002.000000

4.0000006.000000

8.00000010.000000

12.000000

0 0.5 1


E[N

um

be

r o

f W

ait

ing

C

us

tom

ers

]

Threshold

LWISF

Accuracy of Idleness Constraint

1 = 1, 2 = 2, = 1, = 1.5, 2 = 2 = 3, N1=300, N2=200, ¸=674


0.000000

0.200000

0.400000

0.600000

0.800000

1.000000

0.000000 0.500000 1.000000

Predicted

Sim

ula

ted

Threshold

LWISF

Predicted

Summary

• Formulation of the server fairness problem.

• Solution of the approximating diffusion control problem.

• Construction of threshold policy for the original system.

• ²-Threshold Policy (²TP) is ²–asymptotically

optimal.

• TH outperforms LWISF.

Further Research

• Non-ldling assumption• Multi-skill environment• Server Compensation Schemes

Acknowledgement: Rami Atar, Itay Gurvich, Tolga Tezcan & Assaf Zeevi

control for stochastic models via diffusion approximations amy ward, ans lecture series 2008...

Documents