A Comparison and Critique of Eucalyptus, OpenNebula and Nimbus

Peter Sempolinski and Douglas ThainUniversity of Notre Dame

Abstract

Eucalyptus, OpenNebula and Nimbus are three ma-jor open-source cloud-computing software platforms.The overall function of these systems is to manage theprovisioning of virtual machines for a cloud providinginfrastructure-as-a-service. These various open-sourceprojects provide an important alternative for those whodo not wish to use a commercially provided cloud.

We provide a comparison and analysis of each ofthese systems. We begin with a short summary com-paring the current raw feature set of these projects.After that, we deepen our analysis by describing howthese cloud management frameworks relate to the manyother software components required to create a func-tioning cloud computing system. We also analyse theoverall structure of each of these projects and addresshow the differing features and implementations reflectthe different goals of each of these projects. Lastly, wediscuss some of the common challenges that emerge insetting up any of these frameworks and suggest avenuesof further research and development. These include theproblem of fair scheduling in absence of money, evic-tion or preemption, the difficulties of network configu-ration, and the frequent lack of clean abstractions.

1 Introduction

In recent years, due to the high demand for com-puting resources for processing jobs that require hun-dreds or even thousands of cores, different paradigmshave been developed for harnessing the computationalpower of large groups of processors. Depending on thecomputational setting and the problem being solved,different approaches are being taken, including batchjob systems such as SGE [11], cycle scavenging withCondor [27], or virtual machine spawning with Ama-zon’s EC2 cloud [2].

The various definitions of cloud, grid or distributedcomputing are open to some debate. [29] We will con-fine ourselves to considering the idea of infrastructure-

as-a-service, accomplished by providing virtual ma-chines to users. Amazon’s EC2 cloud is arguably oneof the best examples of this paradigm. [2] Of course,this is not to minimize vast varieties of system config-urations that can be referred to as “clouds”. We alsopoint out that Amazon is not the only player in cloudcomputing market, as Sun (Sun Cloud), IBM (BlueCloud), Microsoft (Azure), and many others have theirown systems as well. [7] [13] [21]

In the setting we are considering, a cloud is a groupof machines configured in such a way that an end-usercan request any number of virtual machines (VMs) ofa desired configuration. The cloud will spawn theseVMs somewhere on the physical machines that it owns.The word “cloud” in this context is meant to conveythe semi-ethereal nature of these VMs. The end-userneither knows nor cares where exactly these VMs arephysically located or the configuration of the underly-ing hardware, so long as they can access their bank ofproperly configured VMs. This kind of setup is idealfor applications where a specific hardware configura-tion is needed or users only occasionally need the highcompute capacity.

However, commercial cloud services charge, by thehour, for CPU time. In some settings, such as a largeorganization with many users, it might be more costeffective for the organization to purchase hardware tocreate its own private cloud. This is where open-sourcecloud frameworks such as Eucalyptus,[3] OpenNebula[5] and Nimbus [4] enter the picture. These softwareproducts are designed to allow an organization to setup a private group of machines as their own cloud. Inthis work, we analyze these three open-source cloudframeworks. We selected these three because they rep-resent three different points of interest in the designspace of this particular type of open-source cloud.

In this analysis, we will first briefly address the pre-vious comparisons which have been made of these cloudarchitectures. These previous works largely focused onthe feature set of these clouds. We will briefly summa-rize that work. In this analysis, however, we wish to gobeyond feature comparisons. We will discuss how these

1

software frameworks act as managers that stand in themiddle of a number of other software components. Un-derstanding how these pieces fit together is critical forunderstanding any cloud framework. Next, we will an-alyze how core decisions in the basic architecture andoverall structure of Eucalyptus, OpenNebula and Nim-bus impact the kind of settings and applications forwhich each framework is most suitable. Third, we willidentify several opportunities for improving these soft-ware stacks by identifying some of the challenges thatare common to all three.

2 Previous Work

In looking for comparisons between Eucalyptus,OpenNebula and Nimbus, we found several paperswhich, in some manner, gave comparisons of some as-pects of at least some of these systems. [17], [21], [12],[8], [23], [9] Most of this previous work focused on thefeature set of these software projects. In Table 1, wesummarize the basic feature comparisons from theseprior works.

The open source projects developing Nimbus, Open-Nebula and Eucalyptus are all rapidly expanding. Pre-vious work as recent as 2009 does not include some ofthe new features developed by these projects. For ex-ample, the Nimbus project has recently added supportfor KVM hypervisors, which is not noted in some of theaforementioned citations. In such cases, we update thetable to as recently as possible. However, we we fullyexpect any table of features, including ours, to becomeobsolete. This is especially the case with regard to sup-port for standard API (such as EC2). Moreover, thestandard API themselves often change, rendering priorsupport obsolete. Nevertheless, for features, Table 1is the starting point of our comparison.

3 The Cloud Computing Stack

Before delving into the structures that make Euca-lyptus, OpenNebula and Nimbus unique, we will makea quick overview of the entire cloud computing softwarestack. These three cloud infrastructures only makeup part of the software stack needed to make a cloudcomputing system work. In many ways, software suchas Eucalyptus, OpenNebula and Nimbus stand in themiddle a large number of other components. Further-more, there are often many options for these compo-nents. We include an overview of the software piecesin Figure 1.

In a generic open-source cloud computing system,we can identify six basic components. First, we have

Cloud Controller:Eucalyptus, Open Nebula or

Nimbus

Compute Nodes:

DHCP

hypervisor

VM VM

OS 1

DNS2

3

45

6

Figure 1. Abstract Cloud ArchitectureThe layout of the different parts of a generic cloud com-puting system. Note that the role of the cloud controlsoftware is to coordinate all of these pieces and to pro-vide sufficient abstraction so that a user can simplyrequest VMs with minimal concern for how these com-ponents must interact. The numbers in this diagramcorrespond to the components as we list them in thissection.

hardware and operating systems that are on the vari-ous physical machines in the system. While these mustbe set up properly for any software system, we makeespecial note of the physical hardware and the base op-erating system for two reasons. First, if the processorsof the physical nodes do not have the needed hardwareextensions to run pure virtualization, this limits thesystem to paravirtualization only. While such a cloudsystem can be setup, it greatly limits both the speed ofthe VMs and the flexibility available in choosing soft-ware components for the system. Second, open-sourceframeworks, unlike commercial frameworks, must beflexible enough to work with many underlying systems.(Whereas, commercial clouds only need their system towork with the hardware that they have.)

The second component is the network. This includesthe DNS, DHCP and the subnet organization of thephysical machines. It also includes virtual bridging ofthe network that is required to give each VM a uniquevirtual MAC address. This bridging is accomplishedusing programs such as bridge-utils, iptables orebtables. Moreover, in addition handling the physi-cal nodes, DHCP and DNS processes must be config-ured, in concert with the cloud framework, to handlethe MAC and IP addresses of virtual nodes as well.

2

Table 1. Cloud Architectures ComparedEucalyptus OpenNebula Nimbus

Disk Image Options Options set In private cloud, Depends onby admin most libvirt options left open. configuration

Disk Image Storage Walrus, which A shared file system, Cumulus (recentimitates Amazons S3 by default NFS, or SCP update from GridFTP)

Hypervisors Xen, KVM (VM Ware Xen, KVM, VMware Xen, KVMin non-open source)

Unique Features User management VM migration Nimbus contextweb interface supported broker

The actual setup of these components depends heavilyon whether one is using OpenNebula, Nimbus or Euca-lyptus, as these systems can have different expectationsfor both physical and virtual network configuration.

The third component is the virtual machine hyper-visor, (also known as a Virtual Machine Monitor orVMM). Popular VMMs include Xen and KVM, whichare open-source, VirtualBox, which has an open-sourceversion, and VMware, which is commercial. Theseprograms provide a framework which allows VMs torun. In addition to the actual VMM itself, each ofthese cloud frameworks relies on a library called libvirt,which is designed to provide a facility for controllingthe start and stop of VMs. However, the abstractionis not perfect, as each VMM has unique options thatmust be set. As such, the inputs to libvirt can dif-fer slightly depending on the VMM (and VMM versionused) used. For this and other reasons, the differentcloud frameworks support different subsets of the hy-pervisors.

The fourth component is an archive of VM disk im-ages. In order to usefully run VMs, a virtual hard drivemust be available. In cases where one is simply creat-ing a single VM on a single physical machine, a blankdisk image is created and the VM installs an operatingsystem and other software. However, in a cloud frame-work, where it is expected that hundreds of VMs will beconstructed and torn down in a short amount of time,it is impractical to do a full OS install on each one. Forthis reason, each cloud system has a repository of diskimages that can be copied and used as the basis fornew virtual disks. In any given cloud, we must make adistinction between template disk images and runtimedisk images. The template disk images are those storedin a disk image repository to be used for multiple VMs.When a VM is spawned, one of those templates copiedand is packaged into a disk image appropriate for thegiven hypervisor. Usually, this involves adding a swappartition and padding the disk image to the appropri-ate size. It is the actual runtime image that is used bythe virtual machine.

The fifth component is the front-end for users.There must be some interface for users to request vir-tual machines, specify their parameters, and obtainneeded certificates and credentials in order to log intothe created VMs. Some front-ends perform varioustypes of scheduling by giving users an allotment of re-sources which they are not allowed to exceed. Otherfront-ends implement standard API such as EC2. Wenote that the front-end is one of the most customizablepieces of the entire system.

The last component is the cloud framework it-self, where Eucalyptus, OpenNebula and Nimbus areplaced. This framework processes inputs from thefront-end, retrieves the needed disk images from therepository, signals a VMM to set up a VM and then sig-nals DHCP and IP bridging programs to set up MACand IP addresses for the VM.

4 Recurring Themes

Before proceeding with our comparison, we wishhighlight four main ideas that consistently reoccurwhen looking at these programs. First, is the aboveidea of a complete cloud computing software stack. Acloud control system sits in the middle of a huge num-ber of other components. Indeed, the actual cloud con-troller is only a small part of the overall system. Hav-ing such a high number of software interactions makescompatibility a constant issue. We are also lead, bythis, to our second major theme, which is customiz-ability. This idea is to be expected, given that theseare open-source projects. Part of the appeal of settingup a private cloud, as opposed to using a commercialone, is that the administrator can have more controlover the system. One of the most important ques-tions to be asked about each of these frameworks isthe degree to which customization is allowed for bothadministrators and users. Most notably, support forstandard API interfaces is often one of these customiz-able components. For example, OpenNebula permits afront-end that uses a subset of the EC2 interface as an

3

option, but also lends itself to customized web front-ends, through its XML-RPC interface.

From this, we reach our third idea, which is the de-gree of transparency in the user interface. One of thehallmarks of the cloud idea in the commercial setting isthe “black-box” nature of the system. The individualuser, in a “pure cloud” is not aware of, or cares, where,physically, his VMs are running. In a more custom-made open-source setting, however, opportunities ex-ist for a greater degree of explicit management withregard to the underlying configuration of physical ma-chine and the location of VMs on them. The degree towhich users can be permitted to examine and work onthese underlying components varies among these sys-tems and can often be fine tuned by administratorswho customize the front-end.

Lastly, we note that open-source software frame-works, in many settings, allow for a much closer in-teraction between users and owners of computationalresources. The users of a small to medium-sized com-pany with its own cloud are far more likely to knowand interact with the cloud administrator then if theywere using a commercial cloud. This allows for usersto be granted a certain level of trust that cannot beextended to commercial users. The level of trust al-lowed to users is a key distinction between clouds. Wealso note, however, that this is another example of acustomizable component of some these systems.

5 Open Source Clouds

Earlier we quickly described some of the currentfeatures of Eucalyptus, OpenNebula and Nimbus, andmore such features can be found by checking the doc-umentation of these programs. We will now analyzethe primary differences in the overarching structureand guiding philosophy of these frameworks. This isto achieve two things. First, the feature sets of thesepieces of software change frequently. It is more inter-esting to examine the aspects of these projects that willnot change as rapidly. Second, we wish to shed somelight on the kinds of settings in which each of theseframeworks is more suitable. (In cases in which a com-mercial version of the relevant cloud product exists, weare referring only to the open-source version.)

EucalyptusEucalyptus is designed to be an open-source answer

to the commercial EC2 cloud. Eucalyptus, for its front-end for users, includes a program called euca2ools,which is very similar to Amazon’s EC2 front-end pro-grams. However, some versions of EC2 itself, as well asRightscale are also compatible front-ends. The overalldesign specification of the cloud have been published,

1) Request VM

2) Push Image

Head Node: (cluster and cloud controller)eucalyptus-cloud eucalyptus-cc dhcpd

euca2ools

3) Pad Image 4) Bridge MAC

5) Request Address

7) Log into VM

Compute Nodes:

hypervisor

VM VM

OS

6) Start VM

Figure 2. EucalyptusThe steps for constructing a virtual machine in a con-figuration of Eucalyptus: 1) A user uses the euca2oolsfront-end to request a VM. 2) The VM template diskimage is pushed to a compute node. 3) This disk imageis padded to the correct size and packaged for use by thehypervisor on the compute node. 4) The compute nodesets up network bridging to provide a virtual NIC witha virtual MAC. 5) On the head node the dhcp is setup with the MAC/IP pair. (Note: Technically this isSTATIC mode. But other modes are similar.) 6) VMis spawned on the VMM. 7) The user can now SSHdirectly into the VM.

[20], along with an overview of the most recent ver-sion, [26] and much of the feature documentation isavailable online. [3] In figure 2, we give the steps takenin spawning a VM in a typical installation.

With regard to the overall philosophy underlyingEucalyptus, the system is very much designed for thecorporate enterprise computing setting, a fact con-firmed by the language used on its website and in itsdocumentation. [20] All told, the structure of Euca-lyptus confirms this focus. First, there is a very strongseparation from user-space and admin-space. Root ac-cess is required for everything done by the adminis-trator on the physical machines themselves. Users areonly allowed to access the system via a web interfaceor some type of front-end tools. (The default is for thisEucalyptus’ own euca2ools) With few exceptions, Eu-calyptus attempts to protect users from as many of thecomplexities of the underlying systems as possible. Forauthentication of the euca2ools, users download a zipfile with the needed keys and follow short instructions

4

on how to load them. Once included scripts set upcertain environment variables, the euca2ools will work.Similarly, rather then expose the complexity of diskconfigurations available under libvirt, the administra-tor sets 5 configurations for available processors, mem-ory and hard drive space, and the user must chooseone of these sizes for each of their VM. In this way,users can be more easily protected from the complexunderlying systems.

The software configuration also leans more towarddecentralizing resources, insofar as possible. The sys-tem allows for multiple clusters, such that while thereis a single head node for handling user interfaces, therecan be multiple cluster controllers. This works partic-ularly well if groups of machines in the cloud are phys-ically separated from each other. Furthermore, Euca-lyptus implements a distributed storage system calledWalrus which is designed to imitate Amazon’s S3 dis-tributed storage. Essentially, users are assigned a capfor the amount of Walrus storage they are allowed touse. The storage is separated into structures which, ifneeded, can be distributed throughout the cloud. EachVM’s running disk image, however, is stored locally onthe compute node. This storage mode further increasesthe decentralization by allowing VMs, once started, torun independently of all other machines.

Eucalyptus also assumes that the administrators ofthe machines have some leeway to configure the net-work in accordance with the expectations of one of thenetwork configurations of Eucalyptus. For example, ifthe administrator is using “SYSTEM”, it is assumedthat the external network is configured to accept newrandom MAC addresses and will assign IP addresses tothem. Not all networks, especially secured ones, willnecessarily do this. The other configurations assumethat the cluster controller is allowed to operate its ownDHCP server. Again, not all network setups will likethis. Moreover, all these components must be config-ured with address ranges that are all in agreement witheach other. As such, network configuration can presenta real challenge in many network settings. Eucalyptusworks best when each cluster is its own subnet, withits own reserved address range on the wider network.

The highly decentralized design of Eucalyptus,with multiple clusters, distributed storage, and locallystored running virtual disks, lends itself to a large num-ber of machines. Second, as far as possible, the inter-nal technical details are hidden from users, catering topersons whose primary focus might not be computerscience. The web administration interface caters to set-tings where there is a large number of users to adminis-ter, including built in “signup” features in which usersof the cloud might not know the administrator. In-

ternal maintenance assumes undisputed root access tocloud machines. Optimal network configuration incor-porates the ability to carefully control network addressspaces, and most optimally, setup a dedicated subnetfor Eucalyptus machines. All these features lend them-selves to conditions which are more likely available ina corporate enterprise setting.

OpenNebulaOpenNebula tends to a greater level of centraliza-

tion and customizability (especially for end-users). Thesteps for spawning a VM are shown in Figure 3. Theconfiguration in this figure is the default configurationwhich is documented on their website. [5] However,there is a huge amount of customizability that is per-mitted in OpenNebula. Specifically, the idea of Open-Nebula is a pure private cloud, in which users actuallylog into the head node to access cloud functions. Thisinterface is a wrapper around an XML-RPC interface,which can also be used directly. We note that a front-end interface, such as EC2, can be appended to thisdefault configuration.

The customization available in OpenNebula affectsboth users and administrators. From the administra-tor’s perspective, the most striking customization avail-able is in the shared file system used to store all ofOpenNebula’s files. OpenNebula, by default, uses ashared file system, typically NFS, for all disk imagesfiles and all files for actually running the OpenNebulafunctions. (We note that it can also use simple SCP totransfer files.) The advantage of this is that it exposesmore of the underlying features of libvirt to the cloudusers and administrators. This most notably includesthings such as live VM migration. In practice, we alsofound that this centralization made the system easierto administer.

The customization of OpenNebula also is madeavailable to users if one stays with the default idea ofa private cloud. In order to spawn a VM, the user pro-vides a configuration file containing parameters whichwould be fed into the VMM command line. This allowsfor memory, processor, network and disk resources tobe requested for essentially any configuration. How-ever, the downside to this kind of customizability isthat it is easy for the user to make a mistake. This isespecially the case if the underlying VMM configura-tion is in any way unusual or the network has specialMAC or IP configuration requirements.

OpenNebula is also very centralized, especially inits default configuration with an NFS filesystem. Thecompute nodes do not need a large amount of harddisk resources. However, the NFS, can constitute abottleneck for resources and requires a large amount ofdisk space. At the time of this writing, the OpenNebula

5

1) Log into Head

Head Node: nfsd oned

dhcpd

ssh

3) Pad Image4) oned-

SSHCompute Nodes:

hypervisor

VM VM

OS

5) Bridge MAC

6) Pull Files

2) Request VM

8) Log into VM

*) Request Address

7) Start VM

Figure 3. OpenNebulaThe steps for constructing a VM in a configuration ofOpenNebula: 1) A user uses ssh to login to the headnode. 2) The user uses the onevm command to requesta VM. 3) The VM template disk image is copied anda copy is padded to the correct size and configurationwithin the NFS directory on the head node. 4) The onedprocess on the head node uses ssh to log into a computenode. 5) The compute node sets up network bridgingto provide a virtual NIC with a virtual MAC. 6) Filesneeded by the VMM on the compute node will be pulledto the compute node via the NFS. 7) VM is spawned onthe VMM. 8) The user can now SSH directly into theVM. *) Typically, the dhcp server is handled separatelyfrom the OpenNebula configuration.

documentation recommended 1TB of disk space in theshared file system per 64 cores in the cloud. Of course,NFS can be made more efficient by devoting resourcesto the head node. However, this has a potential to getexpensive, which might not be appropriate for settingsrelying on open-source software.

We point that customization can affect security. Ofcourse, NFS does not encrypt any of its traffic. Thiscan be a serious problem if the network is open topacket sniffing. Of particular note is the fact thatOpenNebula relies on the administrator user of thecloud (default is called oneadmin) to be able to cleanlyssh into any compute node. To insure this, public andprivate keys are placed within the shared NFS direc-tory. Of course, there are multiple network settingswhich can deal with this, such as using another sharedor distributed file system, using the SCP option in-stead, tunneling the NFS connections or arranging a

subnet that outside computers can’t snoop. Neverthe-less, the default arrangements more or less assume afairly trusted setting, such as a private LAN.

All of these design ideas combine to form as sys-tem that (in its default configuration) is most con-ducive to a small to medium sized set of reasonablytrusted users. Moreover, the level of customizationavailable to administrators and users means that thiskind of system works best when for users who knowwhat they are doing from a technical perspective andcan, therefore, take advantage of the added availablefeatures. Alternatively, the administrator can use anoptional front-end like EC2 to protect the users. How-ever, this still requires an administrator be able to cus-tomize the system for it. We also note that this level ofcustomizability is suitable for researchers of computerscience who wish to experiment with combining cloudsystems with other technologies, such as SGE or Con-dor. Other available, non-standard, customizations in-clude scheduling front-ends [25] [24] and sending over-flow cloud requests to Amazon’s EC2 [19].

NimbusThe Nimbus project explicitly advertises itself as a

“science” cloud solution. [4] [15] [14] Nimbus is alsoaffiliated with the Globus Project and uses Globus cre-dentials for the authentication of users. Until recently,Nimbus relied on GridFTP (a Globus project) to beused as a disk image repository. In their newest ver-sion, they are shifting to Cumulus, a system, like Eu-calyptus’ Walrus, compatible with S3. [30] Like Open-Nebula, Nimbus is incredibly customizable. Figure 4shows the steps in spawning a VM in a “typical” con-figuration.

The customization available in Nimbus, however,has a slightly different focus. One reasonable, al-beit imperfect generalization, is that a large numberof the customizations available in OpenNebula per-tain directly to the underlying technical details of VMcreation, whereas for Nimbus these details are moreprotected. OpenNebula basically allows for switch-ing nearly every component, including the underlyingfile system, the DHCP, the front-end. Moreover, inthe default configuration, much of the customizationis available to users and administrators. Nimbus, onthe other hand, leaves most of the customization tothe administrator and not to the user and has severalmore components which are constants. These compo-nents include the image storage, (previously GridFTPand now Cumulus), the use of Globus credentials for alluser authentication, and the requirement that the run-ning Nimbus process can cleanly SSH into all computenodes. Similarly, while Nimbus is very flexible in thenumber and types of virtual networks that can be set

6

1) Request VM

2) nimbusd-SSH

Head Node: nimbusd

cloud-client

4) Pad Image

6) Request Addres

Compute Nodes:

hypervisor

VM VM

OS

3) Push Image

dhcp

5) Bridge MAC

7) Start VM

8) Log into VM

Figure 4. NimbusThe steps for constructing a virtual machine in a con-figuration of Nimbus. 1) A user uses cloud-client torequest a VM. 2) Nimbus will ssh into a compute node.3) The VM template disk image is pushed to the com-pute node. (In the newest versions of Nimbus, this willbe done using a distributed storage similar to S3 andWalrus.) 4) On the compute node, the disk image ispadded to the correct size and configured. 5) The com-pute node sets up network bridging to provide a virtualNIC with a virtual MAC. 6) A dhcp server on the com-pute node is configured with a MAC/IP pair. 7) VMis spawned on the VMM. 8) The user can now SSHdirectly into the VM.

up [16], the underlying physical mechanism for doing sois much less flexible, (at least as of this writing), andinvolves a DHCP server on each compute node withNimbus choosing a random MAC address.

Lastly, among these three pieces of software, Nimbusis the one which probably pays the most attention tocapacity allocation and capacity overflow. The abilityto give different users different lease limits as a meansof scheduling comes standard with Nimbus, whereas itis an add-on for the other two. Second, the idea ofallowing EC2 or another cloud the ability to pick upexcess demand is heavily researched with Nimbus. [18][10] This capacity is similar to previous research into“federated” clouds. [22]

Given all of these ideas, Nimbus sits somewhat inthe middle of Eucalyptus and OpenNebula on the cus-tomization chain. There are a large number of optionsfor user and administrators in deploying the cloud, but

fewer of those options pertain to the nitty-gritty of theunderlying software stack. The security level is slightlyhigher then in OpenNebula, due to the required inte-gration of Globus certificate credentials. A facility isincorporated into the system for sharing capacity be-tween clouds, if desired. However, Nimbus is not soopen as OpenNebula, which exposes large amounts ofthe underlying software in the default “private cloud”configuration. Nimbus‘ level of openness is ideal for thescientific community, which would be most conduciveto sharing cluster time, but most users probably donot want to contend with the oddities of the underly-ing software configuration.

SummaryGenerally speaking, Eucalyptus is geared toward a

private company that wants their own cloud for theirown use and wants to protect themselves from usermalice and mistakes. OpenNebula is geared towardpersons interested in the cloud or VM technology asit own end. Such persons would want a VM sand-box so they can try new and interesting things on thecomputational side. OpenNebula is also ideal for any-one that wants to stand up just a few cloud machinesquickly. Nimbus looks toward the more cooperativescientific community that might be less interested inthe technical internals of the system, but has broadcustomization requirements. Such a community wouldbe more likely to be familiar with the Globus Toolkitand would be more conducive to sharing excess cloudtime. Table 2 summaries some of the general trendsidentified.

Of course, these are generalizations. However, webelieve that they succinctly capture the overall philos-ophy of these projects, if one is aware of their dynamicnature, as well as the many options available. Thesefactors make the situation a little more complex withregard to assessing performance, however. Therefore,the generalization above is for philosophy rather thenperformance of each of these projects.

6 Future Opportunities

In addition to analyzing the kind of settings in whicheach of these projects is most appropriate for cloudmanagement, we wish to identify three opportunitiesthat exist for all of these frameworks.

Scheduling Our first opportunity for research is inthe area of VM scheduling. We note that OpenNebulaand Eucalyptus, as of this writing, in their default con-figurations, do not do any real form of scheduling, inthe sense of negotiating priority for processors. (To beprecise, Eucalyptus does give each users a cap for spacein the Walrus distributed storage.) Rather, if the re-

7

Table 2. Summary of ResultsEucalyptus OpenNebula Nimbus

Philosophy Mimic Amazon Private, highly Cloud resources tailoredEC2 customizable cloud to scientific researchers

Customizability Some for admin, Basically everything Many parts except for imageless for user storage and globus credentials

DHCP On cluster controller Variable On individual compute node

Internal Security Tight. Root required Looser, but can be Fairly tight, unless deployingfor many things. made more tight if needed. a fully private cloud.

User Security Users are given custom credentials User logs into head (unless Users x509 credentialvia a web interface optional front-end used) is registered with cloud

An Ideal Setting Large group of machines for Smaller group of machines Deploy for less to semi-bunch of semi-trusted users for highly trusted users trusted users familiar with x509

Network Issues dhcpd on cluster controller Admin must set manually dhcpd on every nodebut has many options and Nimbus assigns MAC

sources for a requested VM are available, it is allocated,if not, not. Nimbus allows for user to be given a cap onthe number and size of VMs which they are allowed tocreate. Requests that exceed a particular user’s limitare not honored. Of course, a simple front-end doingthe same thing could easily be added to Eucalyptusand OpenNebula. In the case of OpenNebula, we notean active project that does this. [25]

This idea presents a interesting algorithmic problemfor open-source clouds that is not present in commer-cial clouds. For a commercial cloud the variable ofprice can be used to simplify the idea of user prior-ity. A user simply has the resources that they will payfor. If the demand for resources changes, then the pricecan be raised or lowered. For example, Amazon.comallows users to buy “spot instances” which allow usersto specify a price they are willing to pay. When thedemand reaches a low enough point, Amazon.com willstart instances at that price. [1]

However, private clouds do not have the same typeof price variable. Rather, there is a set of users thathave, for some reason or another, access to the system,between which resources must be negotiated. Further-more, this problem is further complicated by the factthat eviction is not an available option. This is in con-trast to high-throughput engines such as Condor. [27]For Condor, the basic mechanism for resolving resourceconflicts it to evict a job. For most grid engines, themechanism is to delay the onset of a job. However,with these cloud systems, the expectation is that theVMs requested will be available for the duration of thescheduler’s lease. A threat that the VM would dis-appear makes the whole system useless. Also, manyapplications (such as MPI jobs) require groups of VMsand it does little good to delay a part of them. As such,the mechanism for private cloud scheduling is the de-cision, yes or no, for a particular requested lease. This

is an online algorithm problem, since the decision mustbe made upon the request, without knowledge of futurerequests. Furthermore, preemption is usually not anoption, because a VM lease, once granted, is assumedgranted for the duration of the lease.

This sets up an intriguing algorithmic problem.What online algorithm can schedule the requests forVM resources, in absence of money, such that com-putational resources remain available for high priorityusers, but resources are also not left idle unnecessarily?This becomes more of an issue the more that users areexpected to be given access to the private cloud. Fur-thermore, the classic methods of eviction and preemp-tion, used for grids and Condor, do not apply in thesame way.

Networking When networking is discussed in thecontext of private clouds, it is usually referring to thevirtual networking of the VM. Nimbus, in particular,has great interest in “One-click clusters,” or groups ofVMs, spawned all at once, complete with a virtual net-work between them. We briefly note the similarity toprevious work on “virtual clusters” [28]. However, animportant contrast is that the virtual clusters are de-signed to run on grids. As such, the physical resourcesgiven to the virtual cluster are not constant and canbe “pulled” into use more dynamically. A “one-clickcluster” in a cloud system is a static group of VMs.This underlies the concept of clouds providing leasesto VMs, rather then job scheduling. There is anotheraspect of networking, however, the interaction betweenthe virtual and physical networks upon which the cloudis run. While all three of these cloud systems do a rea-sonably good job of providing users with the option ofcreating interesting and useful virtual network config-urations, the assumptions made regarding the physicalnetwork configuration, often outside the cloud admin-istrator’s control, varied greatly. While, as mentioned

8

briefly before, the best solution is to give the cloudadministrator full control of the network space wherethe cloud exists, (for example, by providing a separatesubnet) this is not always practical. In practice, wefound that the most frustrating aspect, in every case,of setting up these programs was configuring the cloudcontroller to cooperate with network parameters thatwere not entirely under our control.

Briefly, each of the clouds handles assigning IP ad-dresses to VMs slightly differently. Eucalyptus eitherrelies totally on the external network to assign IPs toany random MAC address (“SYSTEM” mode) or han-dles the assigning of IP addresses through a DHCPserver that is on the cluster controller node. Open-Nebula relies on the cloud administrator to set up theirown DHCP server that will know what to do with theMAC address ranges for which OpenNebula is config-ured. Nimbus has each compute node run a DHCPserver on that compute node for all VMs on that node.This DHCP server is configured by Nimbus such thatthe desired IP is assigned to the matching MAC ad-dress just randomly produced.

It is easy to see the conflicts that can ensue if thenetwork administrator and the cloud administrator arenot the same person. In Eucalyptus, if the routers areconfigured to filter traffic that is not to certain IP ad-dresses, the router tables must be coordinated with Eu-calyptus’ IP/MAC lists. In OpenNebula, the cloud ad-ministrator must be allowed to set up their own DHCPserver and know what IP and MAC address combina-tions will not conflict with the rest of the network. ForNimbus, there is the same filtering problem as in Euca-lyptus, as well as issues that can arise if MAC addressfiltering is used. (Nimbus only creates random MACaddresses.) Also, these problems are for the defaultcloud configurations. The more the cloud customizes,the more one can expect conflicts.

As such, all of these cloud frameworks have the po-tential problem of network conflicts if they are not theonly thing in their own subnets. There are many prob-able solutions to this. First and most simple is, inthe case of enterprise level deployments, only use des-ignated cloud subnets. A second idea is to make thecloud controller programs more flexible, not only withregard to available virtual networks, but also with re-gard to potential interactions with an externally ad-ministered physical network. In any case, making thisproblem easier would remove one of the major obsta-cles to widening the community of users for each ofthese products.

Leaky Abstractions As we stated before, one ofthe main considerations of these three cloud frame-works is the degree to which they separate the user

from the nitty-gritty of the underlying software imple-mentation. However, the case of the VMM is one keyway in which this abstraction is not perfectly main-tained. Specifically, for each of these cloud frameworks,it is necessary to know which VMM (And, in some set-tings, which version of the VMM) is being used in orderto properly configure a VM instance. Even in Euca-lyptus, the sample disk images supplied by Eucalyptuscontain alternate kernels for Xen or KVM.

Additionally, problems can exist because VMM con-figuration details can vary between underlying operat-ing systems. Libvirt, while serving as a suitable com-mon interface for constructing and monitoring VM,does not abstract out these configuration variationsbetween VMM types. As such, users must be madeaware of some aspects of the underlying configurationin order to properly configure VMs. This is more ofthe problem in OpenNebula then in Eucalyptus, since(again, under the default configuration) OpenNebularequires users to supply more of the details of the VM.In Eucalyptus, many, but not all, of these details mustbe set for all users by the administrator.

Practically speaking, these issues can be separatedinto two aspects. First, is the difficulty of constructingdisk images. However, this is not a tremendous issue,given that, basically, the user must know to include axen-enabled kernel in their configuration if they are us-ing xen. Also, the disk image construction difficulty ex-tends beyond the underlying VMM and includes muchmore interesting issues related to producing a soft-ware stack that is both useful and of reasonable size.There is plenty of prior research on this though, onehighly successful attempt to resolve these issues is theCernVM. [6] The second difficulty is, in practice, morecentered on the abstraction issue. Simply put, whenunderlying details of the system affect the configura-tion of higher-up components, it makes it more likelythe user or the cloud administrator will make mistakes.It is this difficulty that presents a challenge of findinga better abstraction for the libvirt layer, so that com-mands to the hypervisor can truly be the same formatfrom cloud setup to cloud setup. A start to such asolution might include making the libvirt layer moreintelligent with regard to translating generic VM in-structions into hypervisor specific instructions. Thiswould, however, require separate libvirt implementa-tions for each hypervisor/OS combination. However,such a layer would make it easier to configure systemswith heterogeneous hardware/hypervisor setups.

9

7 Conclusion

In analyzing these various open-source cloud com-puting frameworks, we find that there are salient philo-sophical differences between them regarding the overallscheme of their design. While the huge amount of cus-tomization possible, as well as the ongoing feature de-velopment of these tools, makes it difficult to make ac-curate absolute statements about them, we have, how-ever, identified certain strong tendencies in the focus ofeach of these products. Moreover, we have pointed outseveral opportunities for improving these frameworks.

8 Acknowledgements

We wish to express our gratitude to Dr. Paul Bren-ner and Mr. Stephen Bogol who assisted us in testingthese software frameworks. We also are grateful to allpersons involved in mailing lists, development and doc-umentation of the Eucalyptus, OpenNebula and Nim-bus projects.

References

[1] Amazon EC2 Spot Instances.http://aws.amazon.com/ec2/spot-instances/.

[2] Amazon elastic compute cloud (EC2).http://aws.amazon.com/ec2/.

[3] Eucalyptus Home Page. http://www.eucalyptus.com/.[4] Nimbus Home Page. http://www.nimbusproject.org/.[5] Open Nebula Home Page. http://www.opennebula.org/.[6] P. Buncic, C. A. Sanchez, J. Blomer, L. Franco, S. Klemer,

and P. Mato. CernVM - a virtual appliance for LHC appli-cations. Proceedings of Science - XII Advanced Computingand Analysis Techniques in Physics Research, November2008.

[7] R. Buyya, C. S. Yeo, S. Venugopal, J. Broberg, andI. Brand. Cloud Computing and Emerging IT Platforms:Vision, Hype, and Reality for Delivering Computing asthe 5th Utility. Future Generation Computer Systems,25(6):599–616, June 2009.

[8] D. Cerbelaud, S. Garg, and J. Huylebroeck. Opening TheClouds: Qualitative Overview of the State-of-the-art OpenSource VM-based Cloud Management Platforms. Proceed-ings of the 10th ACM/IFIP/USENIX International Con-ference on Middleware, (22), 2009.

[9] P. T. Endo, G. E. Goncalves, J. Kelner, and D. Sadok.A Survey on Open-source Cloud Computing Solutions.Brazilian Symposium on Computer Networks and Dis-tributed Systems, May 2010.

[10] G. V. M. Evoy, B. Schulze, and E. L. Garcia. Performanceand Deployment Evaluation of a Parallel Application in anon-premises Cloud Environment. Proceedings of the 7thInternational Workshop on Middleware for Grids, Cloudsand e-Science, 2009.

[11] W. Gentzsch. Sun Grid Engine: Towards Creating a Com-pute Power Grid. Proceedings of the 1st International Sym-posium on Cluster Computing and the Grid, 2001.

[12] P. C. Heckel. Hybrid Clouds: Comparing Cloud Toolkits.2010.

[13] D. Hilley. Cloud Computing: A Taxonomy of Platform andInfrastructure-level Offerings. Technical Report 13, Centerfor Experimental Research in Computer Systems - GeorgiaInstitute of Technology, April 2009.

[14] C. Hoffa, G. Mehta, T. Freeman, E. Deelman, K. Keahey,B. Berriman, and J. Good. On the Use of Cloud Computingfor Scientific Workflows. SWBES 2008, December 2008.

[15] K. Keahey, R. Figueiredo, J. Fortes, T. Freeman, andM. Tsugawa. Science Clouds: Early Experiences in CloudComputing for Scientific Applications, August 2008.

[16] K. Keahey and T. Freeman. Contextualization: ProvidingOne-Click Virtual Clusters. eScience 2008, December 2008.

[17] Z. Lei, B. Zhang, W. Zhang, Q. Li, X. Zhang, and J. Peng.Comparison of Several Cloud Computing Platforms. Sec-ond International Symposium on Information Science andEngineering, pages 23–27, 2009.

[18] P. Marshall, K. Keahey, and T. Freeman. ElasticSite: Using Clouds to Elastically Extend Site Resources.IEEE/ACM International Symposium on Cluster, Cloudand Grid Computing (CCGrid 2010), May 2010.

[19] R. Moreno-Vozmediano, R. S. Montero, and I. M. Llorente.Elastic Management of Cluster-based Services in theCloud. Proceedings of the 1st workshop on Automated con-trol for datacenters and clouds, June 2009.

[20] D. Nurmi, R. Wolski, C. Grzegorczyk, G. Obertelli, S. So-man, L. Youseff, and D. Zagorodnov. The EucalyptusOpen-source Cloud-computing System. 9th IEEE/ACMInternational Symposium on Cluster Computing and theGrid, pages 124–131, 2009.

[21] B. P. Rimal, E. Choi, and I. Lumb. A Taxonomy and Sur-vey of Cloud Computing Systems. Fifth International JointConference on INC, IMS and IDC, pages 44–51, 2009.

[22] B. Rochwerger, D. Breitgand, E. Levy, A. Galis, K. Na-gin, I. M. Llorente, R. Montero, Y. Wolfsthal, E. Elmroth,J. Caceres, M. Ben-Yehuda, W. Emmerich, and F. Galan.The RESERVOIR Model and Architecture for Open Fed-erated Cloud Computing. IBM Systems Journal, 2008.

[23] B. Sotomayor, I. Foster, R. S. Montero, and I. M. Llorente.Virtual Infrastructure Management in Private and HybridClouds. IEEE Internet Computing, 2009.

[24] B. Sotomayor, K. Keahey, and I. Foster. Combining BatchExecution and Leasing Using Virtual Machines. HPDC2008, June 2008.

[25] B. Sotomayor, R. S. Montero, I. M. Llorente, and I. Foster.Capacity Leasing in Cloud Systems using the OpenNebulaEngine. Workshop on Cloud Computing and its Applica-tions 2008 (CCA08), October 2008.

[26] T. Tan and C. Kiddle. An Assessment of Eucalyptus Ver-sion 1.4. Technical Report 2009-928-07, Grid Research Cen-tre, University of Calgary, Canada, 2009.

[27] D. Thain, T. Tannenbaum, and M. Livny. Condor and theGrid. In Grid Computing: making the global infrastructurea reality, pages 299–350. John Wiley and Sons, 2003.

[28] E. Walker, J. P. Gardner, V. Litvin, and E. L. Turner.Personal adaptive clusters as containers for scientific jobs.Cluster Computing, 10(3):339 – 350, September 2007.

[29] L. Wang, G. V. Laszewski, M. Kunze, and J. Tao. CloudComputing: a Perspective Study. Proceedings of theGrid Computing Environments (GCE) workshop, Novem-ber 2008.

[30] L. Wang, J. Tao, M. Kunze, D. Rattu, and A. C. Castel-lanos. The Cumulus Project: Build a Scientific Cloud fora Data Center. In proceedings of International Conferenceof Cloud Computing and Applications, October 2008.

10