1 scheduling i/o in virtual machine monitors© 2008 diego ongaro scheduling i/o in virtual machine...

Scheduling I/O in Virtual Machine Monitors

Diego Ongaro, Alan L. Cox, and Scott RixnerRice UniversityMarch 5, 2008

2Scheduling I/O in Virtual Machine Monitors© 2008 Diego Ongaro

Server Consolidation

Virtual machine monitors enable server consolidation

Consolidated domains have wildly different workloads

Do current solutions actually achieve fairness?

Streaming Media Server

Voice-over-IP Server

Application Server

Web Server

Database Server

. . .

Virtualized Host

Guest Domain 1

Guest Domain 2

Guest Domain 3

Guest Domain 4

. . .

consolidation


Xen Performance

02468

101214161820

Pro

cess

or

Uti

liza

tio

n (

%)

G1 G2 G3 G4 G5 G6 G7

Computation Guests

020406080

100120140160180200

Ban

dw

idth

(M

bp

s)G1 G2 G3 G4 G5 G6 G7

Streaming Guests

Ping latency to G8: 259.2ms Ping latency to G8: 6.6ms

Ping additional idle guest (G8):


Hypervisor – thin virtualization layer Driver Domain – hardware control Guest domains given restricted access

Xen Architecture

Hardware

DriverDomain

Xen ControlSoftware

Guest Domain

UserSoftware

GuestDomain

UserSoftware

GuestDomain

UserSoftware

Xen Hypervisor


Receiving a Network Packet

time

Hardware

DriverDomain

Xen ControlSoftware

Guest Domain

UserSoftware

GuestDomain

UserSoftware

GuestDomain

UserSoftware

Xen Hypervisor

Har

dwar

eIn

terr

upt

Virt

ual

Inte

rrup

t

Driv

er D

omai

n

Targ

et D

omai

n

Sche

dule

r

Sche

dule

r

Virt

ual

Inte

rrup

t

Pack

et A

rriv

es


Scheduling Delays Scheduler has a large effect on latency

As packets arrive, two opportunities to run any number of domains, delaying I/O

Delay proportional to the number of runnable domains

time

Har

dwar

e In

terr

upt

Virt

ual I

nter

rupt

Driv

er D

omai

n

Targ

et D

omai

n

Sche

dule

r

Sche

dule

r

Virt

ual I

nter

rupt

Pack

et A

rriv

es

Oth

er D

omai

ns

Oth

er D

omai

ns


Scheduler Fairness Virtual machine monitors must fairly support

both computation and I/O domains Xen supports computation domains well, but has

mixed results with I/O domains

The scheduler plays a significant role in I/O performance Improvements for I/O are not obvious Some of Xen’s current scheduler optimizations for

I/O are beneficial, others are not


Xen's Credit scheduler

Scheduler “optimizations” for I/O

Conclusions

Outline


Credits are assigned to each domain Approximate the fraction of processor resources

each domain will receive Do not indicate when each domain will receive its

fraction

Scheduler increments/decrements credits Periodically deducts credits from running domain Adds credits when majority of credits in the

system have been consumed

Xen’s Credit Scheduler


Scheduler Operation Domain states

Under – domain has credits remaining Over – domain is over its credit allowance

Domains are run in FIFO order by state Over domains only run if no under domains A domain may run for up to 30ms if it has enough credits

After running, return to the run queue by state Behind all other domains in the same state Regardless of runtime or remaining credits

This approach is biased against I/O

O2 O1

Run Queue

U4 U2U3Next domain

to runU1


Scheduler “Optimizations” for I/O I/O Domain Requirements

Low latency High bandwidth Independent of other domains’ workloads

“Optimizations” Boosting idle domains

Included in Xen Ordering the run queue by credits

Proposed for Xen Tickling the scheduler

Included in Xen


Boosting Idle Domains Initially, scheduler had no special features for

I/O domains

Additional scheduling state: boost Higher priority than under Used when an idle domain is sent a virtual interrupt

With boost

O2 O1 U1U4 U2U3

I1U5

O2 O1 U1U4 U2U3

I1B1

Next domainto run

Next domainto run


Impact of Boost Boost improves latency for I/O domains

0

50

100

150

200

250

300

350

400

Time

La

ten

cy

(m

s)

No Boost Boost


Caveats with Boost Ineffective for I/O domains under heavy loads

Must be idle to receive boost Must block quickly to sustain boost

Effect is negated with many I/O domains

O2 O1 B1B4 B2B3

Next domainto run

O2 O1 U1U4 U2U3

vs.


Ordering the Run Queue I/O domains tend to quickly block

Within each state, sorts domains by credits remaining

Short-running I/O domains are re-inserted near the head of the run queue


Impact of Ordering the Run Queue Ordering the run queue can reduce latency for

I/O domains under larger loads Surprisingly, often complements boost

Configuration Latency CPU Time

Boost,

Not Ordered258.2ms 1.2%

No Boost,

Ordered39.0ms 3.1%

Boost,

Ordered3.5ms 13.3%


Tickling the Scheduler Xen tickles the scheduler after each virtual

interrupt

Tickling can reduce latency Often runs virtual interrupt recipient immediately

time

Har

dwar

e In

terr

upt

Virt

ual I

nter

rupt

Driv

er D

omai

n

Targ

et D

omai

n

Sche

dule

r

Sche

dule

r

Virt

ual I

nter

rupt

Pack

et A

rriv

es

Oth

er D

omai

ns

Oth

er D

omai

ns


Tickling Too Soon? Tickling as virtual interrupts are sent may be

too soon Scheduler is uninformed about upcoming virtual

interrupts Gives arbitrary priority to first domain(s) to be sent

a virtual interrupt

Don’t tickle while sending virtual interrupts All recipient domains are then runnable Scheduler runs the domain with the highest priority


Impact of Tickling

020406080

100120140160180200

Ban

dw

idth

(M

bp

s)G1 G2 G3 G4 G5 G6 G7

No Tickling

020406080

100120140160180200

Ban

dw

idth

(M

bp

s)

G1 G2 G3 G4 G5 G6 G7

Tickling


Transcending Xen Scheduling mixed domains

Native vs. VMM scheduling


Mixed I/O- and compute- domains don’t underutilize resources VMM scheduler can not schedule only the I/O process Low latency for a mixed domain can not be guaranteed

Mixed Domains

Hypervisor

GuestDomain

CPU process

GuestDomain

CPU process

I/O process

GuestDomain

CPU process

Driver Domain

Scheduler


Native vs. VMM Scheduling

Hypervisor

Guest

CPU

Guest

CPU

I/O

Guest

CPU

Scheduler

Driver Domain

?

Native

Scheduler

CPU process

I/O processCPU process

CPU process

Native VMM


Summary of Results Boost reduces latency with few I/O domains

Does little harm in other cases

Ordering the run queue favors short-running I/O domains Often complementary to boost

Tickling reduces latency in some cases Often causes unfairness in bandwidth and latency

I/O-intensive applications need their own domains


Conclusions Scheduling has a large impact on I/O fairness

Can cause exorbitant latency Can cause bandwidth inequity

Xen’s Credit scheduler is not fair for I/O Does well with compute domains No notion of timeliness, which is needed for I/O

Further work needs to be done to solve the mixed domains problem

1 scheduling i/o in virtual machine monitors© 2008 diego ongaro scheduling i/o in virtual machine...

Documents

diego ongaro scheduling

io domains xen

domains slide

virtualized host guest

diego ongaro credits

io conclusions

diego ongaro hypervisor

io performance improvements