the denali project
DESCRIPTION
The Denali Project. Andrew Whitaker , Marianne Shaw, Rick Cox, Steven D. Gribble. http://denali.cs.washington.edu. {andrew,rick,mar,gribble}@cs.washington.edu. A Brief History of VMMs. Introduced by IBM in the late 1960’s: CP/67 (1966) VM/370 (1972) z/VM (2004) - PowerPoint PPT PresentationTRANSCRIPT
The Denali Project
Andrew Whitaker, Marianne Shaw, Rick Cox, Steven D. Gribble
http://denali.cs.washington.edu{andrew,rick,mar,gribble}@cs.washington.edu
A Brief History of VMMs
• Introduced by IBM in the late 1960’s:– CP/67 (1966)
– VM/370 (1972)
– z/VM (2004)
• Found wide use as a time-sharing system system– Ideological competitor to Multics, UNIX
• Today: VMMs have enjoyed a popularity resurgence – Industry: VMWare, Virtual PC (Microsoft), IBM
– Academia: Disco, Denali, ReVirt, Xen
Virtualizable Architectures
• Previous systems focused on strict virtualizability [Goldberg, 74]– Goal: minimize OS
changes
– VMM strictly emulates the phys. hardware
virtual arch
phys arch
hardware
…VM
VMM=
VM VM VM
The Denali Philosophy
• Key insight: software is not hardware– We can modify the VMM to achieve other goals
• Two ways to exploit change:– Modifying the virtual architecture [OSDI 2002]
– Changing the VMM implementation [NSDI 2004]
Outline
• Virtual Machine Monitor Background
• Denali and Architectural Changes– Motivation and approach– Architecture changes– Results
• Denali and Implementation Changes
Motivation
• Growing importance of untrusted code:– Internet services: ASP model, service authors
upload code into hosting center– PlanetLab: “slice” distributed platforms amongst
competing experiments, apps– Your desktop: Java{script}, untrusted binaries,
email attachments
• Needed: strong isolation platform– must be secure, efficient, and extensible
Why we need a new system
• Previous platforms are are inadequate:– Java sandboxes, ActiveX controls, system call
monitoring, virtual private servers, enhanced reference monitors
• Why?– require complex security policy– complexity, wide OS API– performance and/or scaling bottlenecks
Basic approach
• Focus on Internet services– Little sharing required
• Straw man: one service per physical machine– not cost-effective for many
services
• Instead, use virtual machines to emulate separate hardware
hardware
VMM
Web
server
Game
server
Overlay
router
VMMs provide strong isolation
• namespace isolation– no sharing possible, except through network
– eliminates security policy problems
• type-I VMMs run on bare hardware– no layer-below vulnerabilities, unfettered resource
management ability
• no high-level abstractions– eliminates complexity, reduces width of API
But, VMMs do too much!
• VMMs support isolation and full HW emulation– though some VMMs are
moving away from this…
• Isolation kernels separate the goals of isolation and emulation– make a clean break from
the underlying phys arch.
Goal VMM Isolation kernel
strong isolation
full hardware emulation
simplicity, scale, performance
Outline
• Virtual Machine Monitor Background
• Denali and Architectural Changes– Motivation and approach– Architecture changes– Results
• Denali and Implementation Changes
The Denali Virtual Architecture
• Retain x86 instruction set for performance
• Change other aspects to gain simplicity, scale, performance– gives up legacy OS
support
virtual arch
phys arch
hardware
…VM
Denali
VM VM VM
Arch. Changes for Simplicity
• Real hardware is unnecessarily complex
• Denali “scrubs” the hardware architecture:– Omit rarely used features
– Revise other features (interrupts, MMU)
– Streamlined hardware devices• Hardware Ethernet drivers make 12 PIOs per send
• Denali’s Ethernet requires single PIO
Source Code Complexity
0
10000
20000
30000
40000
50000
60000
70000
Linux Denali
Source lines
core kernel
arch. dependent
drivers
0
10000
20000
30000
40000
50000
Linux Denali
Source lines
storage
networking
virtual memory
other
Denali’s core kernel is roughly an order-of-magnitude smaller than bare-bones Linux 2.4.16
a) total lines b) core kernel breakdown
Arch. Change for Scale• Denali exposes an idle-with-timeout instruction
• Avoids a 50% throughput reduction for many VMs
0
100
200
300
400
500
0 50 100 150 200 250 300# virtual machines
Served load (req/sec)
idle-with-timeout
idle-without-timeout
Arch. Change for Performance
• Denali uses asynchronous, batched interrupt delivery
– avoiding context switches results in a 30% performance improvement
VM 1 VM 2 VM 3 VM 4scheduled VM
virtual interrupts 4 4
physical interrupts 4 4 4 4
time
Outline
• Virtual Machine Monitor Background
• Denali and Architectural Changes– Motivation and approach– Architecture changes– Results
• Denali and Implementation Changes
Experimental Setup
• Hardware– Intel Pentium 4 CPU; 1 GB RAM– 3 IDE disks– Intel PRO/1000 gigabit Ethernet NIC
• Software: apps run on Ilwaco guest OS– includes threads package, TCP/IP stack, libc– doesn’t use Denali MMU
• apps linked against Ilwaco like a libOS
Ethernet Virtualization Overhead
• Virtualization overhead is less than 20%
user/kernel crossing 3% , 2%
VM’s device driver
1% , 1%
TCP Stack
37% , 42%
read( )
packet copy VNIC demux Ethernet driver
12% , 13% 5% , 4% 43% , 39%
Ethernetpacketarrival
Breakdown of UDP receive costs (100 byte , 1400 byte packets)
total cost (cycles): 43477, 49002
Web Server Performance
• BSD: one kernel crossing per syscall
• Denali: one kernel crossing per packet
0
100
200
300
400
500
600
1 10 100 1000
document size (KB)
throughput (Mb/s)
BSD Mb/sec
Denali Mb/sec
0
100
200
300
400
500
600
1 10 100 1000document size (KB)
throughput (Mb/s)
BSD Mb/sec
BSD-syscall Mb/sec
Denali Mb/sec
Swapping in Denali
• Mechanism: demand-loading of “virtual” physical memory from statically allocated swap
• Factors that influence performance:– # disks– memory size– popularity distribution– block size– data locality– striping policy
Scale in Denali: 10,000 Web Servers
0
1000
2000
3000
4000
5000
6000
0 200 400 600 800 1000 1200 1400
# VMs
req/sec
0
10
20
30
40
0 2000 4000 6000 8000 10000# VMs
req/sec
In memory performance Disk-bound performance
disk paging bandwidth
• Denali utilizes most of the available hardware resources
memoryexceeded
Outline
• Virtual Machine Monitor Background
• Denali and Architectural Changes– Motivation and approach– Architecture changes– Results
• Denali and Implementation Changes
The VMM as a Service Platform
• Adding services to a VMM:– Migration (Intel Research Pittsburgh, Stanford)
– Intrusion detection (Stanford)
– System replay (Michigan)
– Fault Tolerance (Cornell)
• Advantages:– Virtual devices are relatively simple
– VMM has a “whole-system” perspective
Denali: An Extensible VMM
• Problem: existing VMMs are hard to extend– New services require significant re-implementation or reverse-
engineering
Denali approach:– Factor out VMM design decisions (disk, swap, etc.)
• “Parent” virtual machines can implement functionality on behalf of their “children”
– Encapsulate these decisions with clean interfaces• Supply default implementations for commonly used components
A Denali Interface
// the virtual Disk device callback functionsstruct Disk {
int (*diskWrite)(char *buffer, int offset, int num_sectors);
int (*diskRead)(char *buffer, int offset, int num_sectors);
int (*getSizeInSectors)(void);
int (*getSectorSize)(void);
} Disk;
Migrating an Apache Web Server
0
50
100
150
200
250
300
0 5 10 15 20 25
Time (seconds)
Req/secMachine A
Machine B
Implemented in only 289 lines of C source code
Migration
Conclusions
• Denali exploits the degree-of-freedom offered by software to enhance VMMs– Architecture changes for scale,
performance, simplicity
– Implementation changes for providing virtual machine services