© 2007 ibm corporation ibm t. j. watson research center xen summit 2007 | april 17, 2007 xensocket...
TRANSCRIPT
© 2007 IBM Corporation
IBM T. J. Watson Research Center
Xen Summit 2007 | April 17, 2007
XenSocketSuzanne McIntosh Security, Privacy and Extensible Technologies IBM T. J. Watson Research Center [email protected]
Interdomain transport for VMs
2
McIntosh - IBM T. J. Watson Research Center
XenSocket : Interdomain Transport for VMs | Xen Summit 2007 © 2007 IBM Corporation
IBM T. J. Watson Research CenterXiaolan ZhangPankaj RohatgiSuzanne McIntosh
BAE SystemsJohn Linwood Griffin
We would like to thank the following people:Ronald Perez Douglas Lee Schales Anthony LiguoriRyan Harper Muli Ben-Yehuda Eric Van HensbergenReiner Sailer Stefan Berger Wesley Most
XenSocket Team
3
McIntosh - IBM T. J. Watson Research Center
XenSocket : Interdomain Transport for VMs | Xen Summit 2007 © 2007 IBM Corporation
XenSocket Project – Background
4
McIntosh - IBM T. J. Watson Research Center
XenSocket : Interdomain Transport for VMs | Xen Summit 2007 © 2007 IBM Corporation
Use Xen to enhance security of System S, a stream processing system described as follows:
High-throughput, large-scale, distributed stream processing system Security architecture being developed by IBM Research Extracts important information by analyzing voluminous amounts of
unstructured and mostly irrelevant data Example applications
– Analyze audio, video and data feeds to support trading activities in financial institutions
– Disaster response through analysis of vehicular movements, traffic and other sensors, news reports, etc.
XenSocket Project – The Objective
5
McIntosh - IBM T. J. Watson Research Center
XenSocket : Interdomain Transport for VMs | Xen Summit 2007 © 2007 IBM Corporation
Enhance security with minimal performance hit– Throughput is key metric of goodness for our purposes
– Achieve throughput approaching that of UNIX domain socket
– Especially targeting message sizes in 10 KB – 100 KB range
Facilitate porting of code to the Xen environment– Implement socket-based interface
XenSocket Project – The Requirements
6
McIntosh - IBM T. J. Watson Research Center
XenSocket : Interdomain Transport for VMs | Xen Summit 2007 © 2007 IBM Corporation
Unfortunately, interdomain communication using the Xen virtual network fell short of throughput required for System S.
Inefficient same-system networking performance is a well-known problem wrt VMs [1] [2]
We speculate that the inefficiency can be attributed to
– Overhead incurred by TCP/IP stack
– Repeated hypercalls to invoke Xen page flipping
XenSocket Project – The Problem
7
McIntosh - IBM T. J. Watson Research Center
XenSocket : Interdomain Transport for VMs | Xen Summit 2007 © 2007 IBM Corporation
Proof-of-concept built and tested in Xen 3.0.2.
Bypass use of TCP stack Replace page-flipping with copy Statically allocate memory buffers to be shared between two domains Use sockets-based interface to shared-memory-based transport No modification to Xen or OS required XenSocket compiles into a kernel module
XenSocket Project – The Solution
8
McIntosh - IBM T. J. Watson Research Center
XenSocket : Interdomain Transport for VMs | Xen Summit 2007 © 2007 IBM Corporation
XenSocket Design – Overview
Sockets-based interface to shared memory buffers for domain-to-domain communication
Provides one-way tunnel between sender and receiver domains– Conserves memory in the event only one-way communication
needed
Two types of memory pages shared by each endpoint– Descriptor page (4KB) for storing state and control information
– Buffer pages (multiple 4KB pages) form the circular buffer for writing and reading data
9
McIntosh - IBM T. J. Watson Research Center
XenSocket : Interdomain Transport for VMs | Xen Summit 2007 © 2007 IBM Corporation
XenSocket Architecture
DataPlane
128KB Shared Circular Buffer
(32 4KB pages)
ControlPlane
Shared Descriptor Page
(4KB page)
xen_recvmsg()xen_sendmsg()
Tx Data Rx Data
available_bytes
send_offset
available_bytes
recv_offset
Stream in Stream outSenderApp
ReceiverApp
10
McIntosh - IBM T. J. Watson Research Center
XenSocket : Interdomain Transport for VMs | Xen Summit 2007 © 2007 IBM Corporation
XenSocket Implementation – Receiver
Calls socket() API to create socket
Calls bind() API– Bind the socket to an address
– Allocates physical memory for descriptor page and shared circular buffer
– Returns grant table reference of descriptor page
– Uses sender domain ID to allocate event channel for communication with sender
Calls read() or recv() for receiving data– Blocks until data is detected in circular buffer
11
McIntosh - IBM T. J. Watson Research Center
XenSocket : Interdomain Transport for VMs | Xen Summit 2007 © 2007 IBM Corporation
XenSocket Implementation – Sender
Calls socket() to create a socket
Calls connect()– Uses supplied receiver domain ID and grant table reference of
shared descriptor page
– Maps physical pages of shared circular buffer into the virtual address space of sender
– Establishes other end of event channel to communicate events
Calls send() or write() to transmit data– Blocks if buffer full
12
McIntosh - IBM T. J. Watson Research Center
XenSocket : Interdomain Transport for VMs | Xen Summit 2007 © 2007 IBM Corporation
XenSocket Implementation – Data Transfer Algorithm
Core piece of implementation is an efficient data transfer algorithm
Send and receive algorithms use one shared control variable, available_bytes– Indicates number of bytes available for write in circular buffer
Sender and receiver maintain local read/write offsets into circular buffer– Offsets are not shared
13
McIntosh - IBM T. J. Watson Research Center
XenSocket : Interdomain Transport for VMs | Xen Summit 2007 © 2007 IBM Corporation
XenSocket Performance
XenSocket evaluation environment
IBM HS20 blade with dual 2.8GHz Pentium Xeon processors and 4GB RAM
Used netperf version 2.4.2 as our primary benchmark
All data reported was run on Xen 3.0.2 and Linux version 2.6.16.18
Each test was run 3 times, with the average reported
All experiments were run in single CPU mode with hyper-threading disabled to minimize performance variation
14
McIntosh - IBM T. J. Watson Research Center
XenSocket : Interdomain Transport for VMs | Xen Summit 2007 © 2007 IBM Corporation
XenSocket Performance – Common Message Sizes
Throughput Comparison of XenSocket vs. Unix Domain Socket and TCP for message sizes between 512 Bytes and 16 KB. XenSocket achieves up to 72 times the throughput of standard TCP stream at message size of 16 KB.
15
McIntosh - IBM T. J. Watson Research Center
XenSocket : Interdomain Transport for VMs | Xen Summit 2007 © 2007 IBM Corporation
XenSocket Performance – Larger Message Sizes
Throughput Comparison of XenSocket vs. Unix Domain Socket and TCP for large message sizes. Both XenSocket and Unix Domain Socket see a large drop-off when the message size reaches 512 KB and then stabilize around 5-6 Mb/s. The performance curves invert at message size of 512 KB where XenSocket outperforms Unix Domain Socket.
16
McIntosh - IBM T. J. Watson Research Center
XenSocket : Interdomain Transport for VMs | Xen Summit 2007 © 2007 IBM Corporation
What about performance of TCP under Windows and other OSs?
How does XenSocket perform in current Xen release?
Our design of a XenSocket is a one-way communication pipe between two domains. – Traditional view of a socket is a two-way mechanism
– Alternate design would include variable-size circular buffers with logic capable of adapting the buffer reservation size to actual usage of the buffer
Next Steps
17
McIntosh - IBM T. J. Watson Research Center
XenSocket : Interdomain Transport for VMs | Xen Summit 2007 © 2007 IBM Corporation
1. A. Menon, A. L. Cox, and W. Zwaenepoel. Optimizing network virtualization in Xen. In 2006 USENIX Annual Technical Conference, pages 15–28, Boston, Massachusetts, USA, June 2006.
2. A. Menon, J. R. Santos, Y. Turner, G. J. Janakiraman, and W. Zwaenepoel. Diagnosing performance overheads in the Xen virtual machine environment. In VEE’05: First International Conference on Virtual Execution Environments, pages 13–23, Chicago, Illinois, USA, June 2005.
References
18
McIntosh - IBM T. J. Watson Research Center
XenSocket : Interdomain Transport for VMs | Xen Summit 2007 © 2007 IBM Corporation
Questions?
Suzanne McIntoshSecurity, Privacy and Extensible Technologies
IBM T. J. Watson Research [email protected]