emulab and its lessons and value for a distributed testbed jay lepreau university of utah march 18,...
TRANSCRIPT
Emulaband its lessons and value for
A Distributed Testbed
Jay Lepreau
University of Utah
March 18, 2002
What?
A configurable Internet emulator in a room– Today: 168+160 nodes, 1646 cables, 4x BFS (switch)– virtualizable topology, links, software
Bare hardware with lots of tools: Management Software
An instrument for experimental CS research Universally available to any remote
experimenter Simple to use
Points
Programmable, automated mgmt, complete virtualization:– Qualitatively new environment
Most of it will work in wide area
New Stuff Integrated event system
– Underlying pub/sub system– Integrated into ‘ns’ (statically scheduled)– Start/stop programs– Replayable– Dynamic events– User-accessible
Traffic generation– Automatic, from ns script– New generators:
• TG (tcp, udp)• ‘nse’ with udp, tcp, ftp, telnet
New Stuff (cont’d)
4 node types:– Real, running in the local rack, controlled env.– Real, running ‘nse’– [Simulated]– [Real, in wide-area]
Link configuration and monitoring– Latency, bw, plr, RED, queue size– Link monitoring and capture
GUI network config applet Full-day SIGCOMM tutorial Aug’02
“Programmable Patch Panel”
PCPC
Web/DB/SNMPSwitch MgmtUsers
Internet
Control Switch/Router
Serial
Sharks Sharks
160168
PowerCntl
Fundamental Leverage:
Extremely ConfigurableEasy to Use
– Power– Performance– Virtualization
Key Design Aspects Allow experimenter complete control
– Configurable link bandwidth, latency, and loss rates, via transparently interposed “traffic shaping” nodes that provide WAN emulation
… but provide fast tools for common cases– OS’s, state mgmt tools, IP, batch, ...– Disk loading – 6GB disk image FreeBSD+Linux
• Unicast tool: 88 seconds to load• Multicast tool: 40 nodes simultaneously in < 5 minutes
Virtualization– of all experimenter-visible resources– node names, network interface names, network
addrs– Allows swapin/swapout, easily scriptable
Key Design Aspects (cont’d)Flexible, extensible, powerful
allocation algorithm– Matches desired “virtual” topology to
currently available physical resourcesPersistent state maintenance:
– none on nodes, all in database– work from known state at boot time
Familiar, powerful, extensible configuration language: ns
Separate, isolated control network
Lessons for wide area testbed
Central control: at this scale (1000s) it’s easy
Database!Control node for each site: great
benefits, cheap marginal cost– Trusted, firewall, local disk cache, power
control, console line
Ease of use is dominant driver
Lessons… Generalized resource alloc/mapping
algorithm is great (eg, vs Grid) Get it going quickly, keep it going while
add new stuff– Like a startup– Use feedback and demand– 2.5 years in
Simple authorization model Most of our model and code will work in
wide-area
Lessons…
Freedom for users is freedom for the management software and people
“You’ve got root, use it.”
Over-provision
FreeBSD Jail, or Eclipse/BSD, or VMWare, or ….
Testing is tricky
Have real hardware that can’t virtualize
Test suite part of build Clone DB works some… 8-node minibed Nightly regression testing Schema evolution script/diff/check Developers use/test 3 diff. browsers
Code Base Today 24,100 Web front end 23,900 Back end 2000 ns front end 4200 Resource mapping 4900 Diskimg compression/casting/load 8400 Scripts/daemons from nodes to DB 5000 Event system 6200 Remote console interaction/logging 3300 Regression testing harness and tests 700 Node health monitoring 3700 Documention of internals
More stats
21 “programs”318 “scripts” (including 90 php
scripts, 71 small boot-time scripts)35% Perl32% C19% php12% html, Java, tcl, other
The Database Today
Started with ~18 tables54 tables, 413 columnsGeneral categories
– Physical world: 11 tables, 65 cols– Virtual world: 7 tables, 83 cols– Operational state: 22 tables, 180 cols– Admin data: 14 tables, 85 cols
Note how much operational state shows how much work needs to be done
Testbed Users
30 active projects– more registered– 25 External– About 40/30/30%
dist sys/activenets/traditional networking
~110 users 990 “experiments” in last 8 months
• 7.5/day recently• 40% testbed development
More Sites
More emulab’s under construction:– Kentucky– Umass– Duke, CMU, Cornell, Stuttgart– Others stated intent:
MIT, WUSTL, Princeton, HPLabs, Intel/UCB, Mt. Holyoke, …
Ongoing and Future Work
Federation– heteregeneous sites– resource allocation
Wireless nodes, mobile nodes IXP1200 nodes, tools, code fragments
– Routers, high-capacity shapers Simulation/emulation transparency Event system Scheduling system Topology generation tools and GUI Data capture, logging, visualization tools Microsoft OSs, high speed links, more nodes!
A Global-scale Testbed Federation key Bottom-up “organic” growth
– Local autonomy and priority– Existing hardware resources– Provides diverse hardware
• PCs• Wireless, mobile• Real routers, switches (Wisconsin, …)• Network processors (IXP’s)• Research switches (WUSTL)
But, top-down is much easier: a good start
NSF ITR Proposal (Nov 01)
Global-scale testbedUtah primary
– Research emphasis: software component for heterogeneity; resource allocation/mapping
Collaborators:– Brown, co-PI (resource allocation)– MIT (RON overlay, wireless)– Duke (ModelNet muxing, early adopter)– Mt. Holyoke (education)
Types of Sites
High-end facilitiesGeneric clustersGeneric labs“Virtual machines”
Internet2 links between some sites
Result…
Loosely coupled distributed system
Controlled isolation
“Internet Petri Dish”
New Stuff: Extending to Wireless and
MobileProblems with existing approaches
Same problems as wired domain But worse (simulation scaling, ...) And more (no models for new technologies, ...)
Wireless Virtual to Physical Mapping
Available for universities, labs, and
companies, for research and teaching,
at:
www.emulab.net
A Few Research Issues and Challenges
Network management of unknown and untrusted entities
Security (root!) Scheduling of experiments Calibration, validation, and scaling Artifact detection and control NP-hard virtual --> physical mapping
problem Providing a reasonable user interface ….
How To Use It ... Submit ns script or GUI via web form Behind the scenes:
– Generates config from script & stores in DB– Maps specified virtual topology to physical nodes– Allocate resources– Provides user accounts for node access– Assigns IP addresses and host names– Configures VLANs– Loads disks, reboots nodes, configures Oss– Starts event system, traffic generators, link monitoring/control– Yet more odds and ends ...– User does his/her experiment– [Reports results if batch]
Takes ~3 min to set up 25 nodes, 5 secs/node
An “Experiment”
emulab’s central operational entity Directly generated by an ns script, … then represented entirely by
database state
Steps: Web, compile ns script, map, allocate, provide access, assign IP addrs, host names, configure VLANs, load disks, reboot, configure OS’s, run, report