Download - DOE Magellan OpenStack user story
Magellan Experiences with OpenStack
Narayan [email protected] National Lab
The Challenge of High Performance Computing
Scientific progress is predicated on the use of computational models, simulation, or large scale data analysis– Conceptually similar to (or enabling of) traditional experiments
Progress is also limited by the computational capacities usable by applications Applications often use large quantities of resources
– 100s to 100000s of processors in concert– High bandwidth network links– Low latency communication between processors– Massive data sets
Largest problems often ride the ragged edge of available resources– Inefficiency reduces scope and efficacy of computational approaches to tackle particular
large scale problems Historically driven by applications, not services
The Technical Computing Bottleneck
DOE Magellan Project (2009-2011)
Joint project between Argonne and Berkeley Labs ARRA Funded Goal: To assess “cloud” approaches for mid-range technical computing
– Comparison of private/public clouds to HPC systems – Evaluation of Hadoop for scientific computing– Application performance comparison– User productivity assessment
Approach: Build a system with an HPC configuration, but operate as a private cloud– 504 IBM Idataplex nodes– 200 IBM 3650 Storage nodes (8 Disk, 4 ssd)– 12 HP 1TB Memory nodes– 133 Nvidia Fermi nodes– QDR Infiniband– Connected to the ESNet Research Network
Initial Approach
Setup Magellan as a testbed– Several hardware types, many software configurations
Chose Eucalyptus 1.6 as the cloud software stack– Mindshare leader in 2009– Had previous deployment experience– Supported widest range of EC2 Apis at the time
Planned to deploy 500 nodes into the private cloud portion of the system– Bare metal provisioning for the rest, due to lack of virtualization support for GPUs, etc
Initial Results
Detailed Initial Experiences (2009-2010)
Had serious stability and scalability problems once we hit 84 nodes Eucalyptus showed its research project heritage
– Implemented in multiple languages– Questionable architecture decisions
Managed to get system into usable state, but barely Began evaluating potential replacements (11/2010)
– Eucalyptus 2.0– Nimbus– Openstack (Bexar+)
Evaluation Results
Eucalyptus 2.0 better, but more of the same Openstack fared much better
– Poor documentation– Solid architecture– Good scalability– High quality code
• Good enough to function as documentation surrogate in many cases– Amazing community
• (Thanks Vish!)
Decided to deploy Openstack Nova in 1/2011– Started with Cactus beta codebase and tracked changes through release– By February, we had deployed 168 nodes and began moving users over– Turned off old system by 3/2011 – Scaled to 336 than 420 nodes over the following few months
Early Openstack Compute Operational Experiences Cactus
– Our configuration was unusual, due to scale • Multiple network servers• Splitting services out to individual service nodes
– Once things were setup, the system mainly ran– Little administrative intervention required to keep the system running
User productivity– Most scientific users aren’t used to managing systems– Typical usage model is application, not service centric– Private cloud model has a higher barrier to entry– Model also enabled aggressive disintermediation, which users liked– It also turned out there was a substantial unmet demand for services in scientific
computing Due to the user productivity benefits, we decided to transition the system to
production at the end of the testbed project, in support of the DOE Systems Biology Knowledgebase project
Enable DOE Mission Science
Microbes
Plants
Communities
Transitioning into Production (11/2011)
Production meant new priorities– Stability– Serviceability – Performance
And a new operation team Initial build based on Diablo
– Nova– Glance– Keystone*– Horizon*
Started to develop operational processes– Maintenance– Troubleshooting– Appropriate monitoring
Performed a full software stack shakedown– Scaled rack by rack up to 504 compute nodes
Vanilla system ready by late 12/2011
Building Towards HPC Efficiency
HPC platforms target peak performance– Virtualization is not a natural choice
How close can we get to HPC performance while maintaining cloud feature benefits?
Several major areas of concern– Storage I/O– Network Bandwidth– Network latency– Driver support for accelerators/GPUs
Goal is to build multi-tenant, on demand high performance computational infrastructure– Support wide area data movement– Large scale computations– Scalable services hosting bio-informatics data integrations
Network Performance Expedition
Goal: To determine the limits of Openstack infrastructure for wide area network transfers– Want small numbers of large flows as opposed to large numbers of slow flows
Built a new Essex test deployment– 15 compute nodes, with 1x10GE link each– Had 15 more in reserve– Expected to need 20 nodes– KVM hypervisor
Used FlatManager network setup– Multi-host configuration– Each hypervisor ran ethernet bridging and ip firewalling for its guest(s)
Nodes connected to the DOE ESNet Advanced Networking Initiative
ESNet Advanced Networking Infrastructure
Setup and Tuning
Standard instance type– 8 vcpus– 4 vnics bridged to the same 10GE ethernet– virtio
Standard tuning for wide area high bandwidth transfers– Jumbo frames (9K MTU)– Increased TX queue length on the hypervisor– Buffer sizes on the guest– 32-64 MB window size on the guest– Fasterdata.es.net rocks!
Remote data sinks– 3 nodes with 4x10GE– No virtualization
Settled on 10 VMs for testing– 4 TCP flows each (ANL -> LBL)– Memory to memory
Network Performance Results
Results and comments
95 gigabit consistently– 98 peak!– ~12 GB/s across 50 ms latency!
Single node performance was way higher than we expected– CPU utilization even suggests we could handle more bandwidth (5-10 more?)– Might be able to improve more with EoIB or SR-IOV
Single stream performance was worse than native– Topped out at 3.5-4 gigabits
Exotic tuning wasn’t really required Openstack performed beautifully
– Was able to cleanly configure this networking setup– All of the APIs are usable in their intended ways– No duct tape involved!
Conclusions
Openstack has been a key enabler of on demand computing for us– Even in technical computing, where these techniques are less common
Openstack is definitely ready for prime time– Even supports crazy experimentation
Experimental results shows that on demand high bandwidth data transfers are feasible– Our next step is to build openstack storage that can source/sink that data rate
Eventually, multi-tenancy data transfer infrastructure will be possible This is just one example of the potential of mixed cloud/HPC systems
Acknowledgements
Argonne Team– Jason Hedden– Linda Winkler
ESNet– Jon Dugan– Brian Tierney– Patrick Dorn– Chris Tracy
Original Magellan Team• Susan Coghlan• Adam Scovel• Piotr Zbiegel• Rick Bradshaw• Anping Liu• Ed Holohan