updating xen for the client environment · 2011-02-28 · 11 12/3/07 •xen hypervisor loaded at...
TRANSCRIPT
![Page 1: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/1.jpg)
Updating Xen for the ClientEnvironment
Junhong Jiang, Kevin Tian,Chris Wright, Don Dugger
![Page 2: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/2.jpg)
12/3/072
Legal Content
INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTIONWITH INTEL® PRODUCTS. EXCEPT AS PROVIDED IN INTEL'STERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTELASSUMES NO LIABILITY WHATSOEVER, AND INTEL DISCLAIMSANY EXPRESS OR IMPLIED WARRANTY RELATING TO SALEAND/OR USE OF INTEL PRODUCTS, INCLUDING LIABILITY ORWARRANTIES RELATING TO FITNESS FOR A PARTICULARPURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANYPATENT, COPYRIGHT, OR OTHER INTELLECTUAL PROPERTYRIGHT.
Intel may make changes to specifications, product descriptions,and plans at any time, without notice.
Intel is a trademark of Intel Corporation in the U.S. and othercountries.
Copyright © 2007, Intel Corporation. All rights are protected.
![Page 3: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/3.jpg)
12/3/073
Agenda
• Xen Client Overview
• Boot Scheme/Verified Launch
• Device Virtualization
• Power Management
![Page 4: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/4.jpg)
12/3/074
Glossary
• VA Virtual Appliance
• Xen Client is a platform for implementing VAs
• UOS User Operating System/User OS/Primary User Partition
• SOS Service OS/VA Partition
• VM Virtual Machine/Guest Partition
• VMM Virtual Machine Monitor/Hypervisor
![Page 5: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/5.jpg)
12/3/075
Xen Client
• Xen based virtual appliance infrastructure Limited device virtualization/isolation support
• Single SOS Linux kernel (Para-virtualized) No direct disk access at run time
• Permanent storage provided by UOS• Exception during boot process
•Dom0 provides disk access
• Target platform TXT support
• Measure and authenticated boot• Trust chain from BIOS
VT-d Support
• Virtualizes NIC TPM
![Page 6: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/6.jpg)
12/3/076
Architecture Overview
• VMM Xen hypervisor
• 64 bits Dom0
• Para-virtualized Linux.• Runs out of RamFS• Minimal components
•Control Panel, device models, default power management policy, virtualappliance specific agents
• Headless•Capable of booting messages
• UP
• Service OS Paravirtualized XenLinux
• Fully virtualized possibility for future Headless
• Boot up display for SOS in Dom0 Survive even UOS hangs Support single or multiple SOS’s (At least one)
![Page 7: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/7.jpg)
12/3/077
Architecture Overview (Cont.)
• SOS0 (LAN filtering) LAN Filtering
• Owns physical NIC• Provide VNIF BE driver
SMP capable
• SOS1 (VoIP) Own Audio SMP Capable
• User Operating System Near native performance Own rest physical resources (CPU, memory, devices) Virtual LAN
• VNIF front-end (FE) driver SMP Windows XP SP2 (32-bit PAE w/ XD support) Vista (32 and 64-bit) Windows PE Linux
![Page 8: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/8.jpg)
12/3/078
Xen Client Architecture OverviewV
eri
fied
boot
vP
RO
Clie
nt C
fg./
Lau
nch
Pro
vis
ion
/In
stall
Ag
en
t P
rese
nce
Dom0Dom0
Vt-x LT VT-d EM64T Channel ACPI/PM
Shadow PT PIC/PIT Models
Xen hypervisorXen hypervisor
SOSSOS
Netw
ork
Dri
vers
VA
Serv
ices
Ag
en
ts
VA
Dri
vers
VA
Pro
vis
ion
ing
Ag
en
ts
VA
N/W
Rou
tin
g S
rvc.
VA
Sto
rag
e S
erv
ices
Oth
er
VA
Dri
vers
ISV Apps.ISV Apps.
UOSUOS
User Apps.User Apps.
VA
Dri
vers
VA
Serv
ices
Ag
en
ts
Nati
ve D
rivers
VA
Pro
vis
ion
ing
Ag
en
ts
VA
Sto
rag
e S
erv
ices
Oth
er
VA
Dri
vers
TPM HW LAN/WAN SATA, USB, etc.
vTP
M
![Page 9: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/9.jpg)
12/3/079
Disk Layout
• Reserved SOS partition Type 0x71 Pointed to by MBR Contains boot/persistent storage file system
• Boot FS in SOS partition Contains grub loaded modules
• Xen kernel image• Dom0 vmlinuz• Dom0 initrd
SOS modules are not loaded by grub• Dom0 mounts the boot FS• SOS kernel/initrd loaded by domain builder in Dom0
![Page 10: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/10.jpg)
12/3/0710
Disk layout
Part 3
Part 2
Part 1
Part 4 (type = 71)
UOS MBR
Boot Block
UOS Partition
SOS Boot Block
PartitionTableMaster Boot
RecordBoot code
PhysicalDisk
UOSusableDisk
SOSpartition
∞
0
SOS Partition
Part 3
Part 2
Part 1
Part 4 (type = 71)
![Page 11: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/11.jpg)
12/3/0711
• Xen hypervisor loaded at fixed memory Affects minimum memory required
• Dom0 allocated at top of memory First partition loaded
• SOS’s loaded sequentially below Dom0
• UOS loaded at low memory Allows 1-1 guest physical to machine physical mapping
• Needed to execute platform BIOS, especially SMI’s Size of UOS dynamically determined
• Max memory minus fixed size for Xen, Dom0 & SOS’s
• Implications: UOS given contiguous machine physical address
• Hole in memory for Xen Xen hypervisor memory allocation change
Memory Layout
![Page 12: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/12.jpg)
12/3/0712
Physical RAM layout – initial boot
0
640K
1M
256M
Max
BIOS
Grub Modules
Dom0 initrd
Dom0 image
Xen VMM image
![Page 13: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/13.jpg)
12/3/0713
Physical RAM layout - runtime
0
640K
1M
256M
Max
BIOS
Xen VMM image
Dom0 RAM
SOS RAM
UOSgpfn == mpfn
UOSgpfn == mpfn
![Page 14: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/14.jpg)
12/3/0714
Boot sequence – 4 phases
1. Grub loads Xen & Dom0 into memory
2. Dom0 starts
3. SOS starts
4. UOS starts
![Page 15: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/15.jpg)
12/3/0715
Persistent Storage
• SOS runs out of RAM Needs access to persistent storage on disk
• SOS run a disk Front End driver Talks to disk Back End driver in Dom0 during boot Unmounts disk from Dom0 Shuts down disk FE from Dom0 Starts disk FE, now talking to UOS Remounts disk
![Page 16: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/16.jpg)
12/3/0716
Verified Launch – Dom0Dom0 (Linux)
Standard Linux Drivers(VGA, Chipset NV, Disk)
Device Model(PCI Config,
Serial, ICH,..)
Domain Builder(xm/libxc
equivalent)
CMFM
InterVMcommunication driver
vTPM
VM PowerManager
Back-end diskdriver
Xen hypervisorXen hypervisor
SATA, USB, etc.Shared HWUser OptimizedHW
TPMDriver
Kern
elU
ser
![Page 17: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/17.jpg)
12/3/0717
Verified Launch – SOS
SATA, USB, etc.Shared HWUser OptimizedHW
Dom0 (Linux)
Standard Linux Drivers(VGA, Chipset NV,
Disk)
Device Model(PCI Config,
Serial, ICH,..)
Domain Builder(xm/libxc
equivalent)
CMFMvTPM
VM PowerManager
BE diskdriver
Xen hypervisorXen hypervisor
TPMDriver
Linux NICDriver
SOS
Libxc (user levelevent channel)
Back-endNIC driver
vTPM FEDriver
CMFM agent(default)
InterVMcommunication
driver
Front-endDisk
Driver
VA services
InterVMCommunication
driverBE TPM
driver
![Page 18: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/18.jpg)
12/3/0718
Verified Launch – UOS
SATA, USB, etc.Shared HWUser OptimizedHW
Dom0 (Linux)
Device Model(PCI Config,
Serial, ICH,..)
Domain Builder(xm/libxc
equivalent)
CMFMvTPM
VM PowerManager
Xen hypervisorXen hypervisor
TPMDriver
Linux NICDriver
SOS (Linux)
Libxc (user levelevent channel)
Back-endNIC driver
CMFM agent(default)
InterVMCommunication
driver
Front-endDisk
Driver
BE TPMdriver
UOS (Windows)
FE NICdriver
StandardWindows
DrivervTPM FE
Driver
VA services
InterVMCommunication
driver
VAservices
PS agent: Back-endDisk Driver
TPMDriver
InterVMCommunication
driver
Libxc (user levelevent channel)
DeviceModel
![Page 19: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/19.jpg)
12/3/0719
I/O Device Handling
• Default is devices handled directly by UOS Near native performance DMA support
• VT-d provides protection between guests• Non-VT-d platforms will depend upon `well behaved’ guests
• Special devices paravirtualized through SOS LAN Disk
• Xen Client Currently•UOS owned
• Xen Client Future•PCI IOV w/ secure LBA allows dual ownership
Audio• Xen Client Future
•PCI IOV provides secure sharing
![Page 20: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/20.jpg)
12/3/0720
PCI Config Space Virtualization
Dom0Dom0
Agent Integrity
Boot and Startup
ProvisioningVt-x LT VT-d EM64T Channel ACPI/PM
Shadow PT PIC/PIT Models
Xen hypervisorXen hypervisor
SOSSOS
Netw
ork
Dri
vers
VA
Serv
ices
Ag
en
ts
VA
Dri
vers
VA
Pro
vis
ion
ing
Ag
en
ts
VA
N/W
Rou
tin
g S
rvc.
VA
Sto
rag
e S
erv
ices
Oth
er
VA
Dri
vers
ISV Apps.ISV Apps.User Apps.User Apps.
VA
Dri
vers
VA
Serv
ices
Ag
en
ts
Nati
ve D
rivers
VA
Pro
vis
ion
ing
Ag
en
ts
VA
Sto
rag
e S
erv
ices
Oth
er
VA
Dri
vers
TPM HW LAN/WAN SATA, USB, etc.
TPMDriver
ICH(PowerControl)
PCI Configspace
Device ModelsDevice Models
vTPM
UOSUOS
![Page 21: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/21.jpg)
12/3/0721
I/O Device virtualization - PCI Handling
• Dom0 owns PCI bus enumeration SOS/UOS PCI bus enumeration is virtualized in its host domain.
• SOS vPCI bus enumeration in dom0• UOS vPCI bus enumeration in Dom0
•Policy of device assignment to UOS, other SOSes setup by managementpartition
•Virtualized devices are inserted into PCI hierarchy
• PCI device physical BAR = vBAR in UOS Help for PCI hot plug support and thus docking
![Page 22: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/22.jpg)
12/3/0722
Device assignment – PCI hierarchy
• PV (S0S) Virtual PCI front end/back end frame
• Qemu based partition (HVM SOS) Using current Piix4 in qemu as virtual PCI hierarchy Attach assigned physical device to virtual PCI bus in qemu, like current VT-
d effort• Qemu maintains vCONFIG_SPACE to pCONFIG_SPACE mapping• See PCI express, depends on virtual chipset in qemu
• Native based partition (UOS) See exactly same PCI hierarchy as physical
• FE becomes the placeholder of detached devices Need a stub to maintain vCONFIG_SPACE to pCONFIG_SPACE mapping
• HVM PCI config space interception Trapped by Xen, and emulated by vPCI Device Model
• Handlers has filter to decide which field can be written through• Handlers further issues io/mmio/interrupt assignment based on captured info
![Page 23: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/23.jpg)
12/3/0723
Xen Client: Device Assignment
• “Hide” all PCI devices except for the Dom0 assigneddevices from Dom0 device drivers
• Assign PCI devices to guest
• Attach the assigned device to QEMU vPCI bus
• Assign corresponding VT-d context entry to guestdomain
• Intercept PCI config access in Xen and Qemuappropriately
• Turn on VMCS IOPORT bit to allow IOPORT access toassigned devices
• Install P2M entry for MMIO access of the assigneddevice
• Xen intercepts physical device interrupts and re-injectsto the target guest domain with vIRQ
![Page 24: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/24.jpg)
12/3/0724
Platform ACPI Component
Xen Client PM Architecture Overview
DOM0 PM
OSPM Driver
DOM0
VMPM (VM Power Manager)
Semi-Virtual Platform
PlatformACPI Component
UOS PM
TPM Driver
TPM HW
Registers BIOS Tables
Virtual PM ops Real PM opsVM exit/entryXen event
Shared HWUser OptimizedHW
SegregatedHW
PlatformHardware
FEBE FE
ACPIDriver/AMLInterpreter
VA Agent
ACPI Virtualization
PM Event Coordination PCI PM Virtualization
PV SOS(No PM)
SOS Device PM
Use at boot time only
Device Driver
Event notification (via evtchn driver)
Primary UOS
![Page 25: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/25.jpg)
12/3/0725
Power Management
• UOS OSPM policy virtualized
• Policy defined by UOS and enforced by Dom0 UOS controls native devices SOS controls everything else
• Shared devices• CPU• Platform
UOS initiates PM policy (Suspend, Hibernate, C-state or P-statechanges)
![Page 26: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/26.jpg)
12/3/0726
PM - Suspend
• UOS owns physical PCI bridges It will suspend bridges before SOS can suspend it’s devices Implication is that UOS must see virtualized bridges
• One flat PCI space•Could be issue with drivers assuming bus limitations
• Virtual copy of actual topology•Preferable solution
• Suspend UOS suspends devices
• Including virtualized devices (as requested by device D3 request) UOS suspends PCI bridges
• Including any virtualized bridges and conforming to the policy setup inDom0 (VMPM – VM power management)
UOS completes suspend and request ACPI suspend request (ICHI/O port write)
• ICH PM suspend requests suspend of SOSes if needed•Should avoid keeping any state in SOS/DOM0 for restart of SOS/DOM0
• Wake from Suspend Restarts from Xen in real mode and execute
![Page 27: Updating Xen for the Client Environment · 2011-02-28 · 11 12/3/07 •Xen hypervisor loaded at fixed memory Affects minimum memory required •Dom0 allocated at top of memory First](https://reader034.vdocuments.us/reader034/viewer/2022050412/5f89096d7492e50466653173/html5/thumbnails/27.jpg)
12/3/0727
Conclusion
• This is real, download the Alpha release at
http://eit.et.redhat.com
• Questions? Comments? Epithets?