balc cloud - terena · outside world and to other countries. [e.g. right now we are buying hardware...

AakeEdlund,PhDKTHandSNIC,Sweden

NorthernEuropeCloud(ProjectManager)NEW

Bal4cGrid(ProjectDirector)Bal4cGridInnova4onLabandBal4cCloud(Ini<ator,Manager)

Bal4cCloud

14thTerenaTFStorageMee<ng,

Copenhagen,2009.Sept.15,Bal<cCloud

2

–  Started1stofMay,2008,dura<on‐2years

–  Con<nuestheBal<cGridproject(2005‐2008)

–  13partnersfromBal<cStates,Belarus,Poland,Sweden,Switzerland

–  Financing–3MEUR

–  Collabora<ngprojectforEGEE

–  Suppor<ngNGIsandtheircollabora<on

Normalizedcompu<nghoursfortheBal<cGridprojectandthatofthefirstsixmonthsofBal<cGrid‐II

4thTerenaTFStorageMee<ng,Copenhagen,2009.Sept.15,Bal<cCloud

Bal4cGrid‐the

Founda4on

Mo4va4onfor

Bal4cCloud

Whenwestarted,cloudwasconsideredimmature,andonlyforstartupsandpossiblyprototyping.

Thissuiteduswell,knowingthatwehadnotbeenverysuccessfulina\rac<ngbiggercompanies.

Startupandprototypingfocus



•  Ini<atedinJanuary20,2009

•  Aprogramforearlystagecompaniesinourregion,

•  byinvi<ngthemtoahands‐oncourseingridandcloudcompu<ng.–  Acoursepreparedandrunbyus,onourresourcesaswellasondonateddito

frome.g.Amazon.

•  Theaimwiththecourseistohelpearlystagehigh‐techInternetbasedcompaniestotrytheirservicesonnewpla_orms,resul<nginearlyproofofconceptsandlaterexploita<onofgridandcloudintheregion.

•  Ontopofthecoursewe'llbuildanetworkofinnova<vecompaniesintheregion.”



TheVision

AnewwaytoaGractstartups

bgin.wordpress.com

Mo4va4onfor

Bal4cCloud

‐  Lownumberofapplica<onsandapplica<onareasinBG.

‐  Needofelas<cy‐volumeondemand‐awaytohandlecapacityonthefly,tobalancetheusage

‐  Toosteeplearningcurveforgridusers

‐  Lowacceptancewithinindustry–sofar‐  Complicatedtoadaptnewapplica<ons,typically,needa“project”

todothat

Complementtocurrentgridservices



Mo4va4onfor

Bal4cCloud

Onemore,hopefully,easieraccesspointtoourBal<cGridresources.

So,onemoreaccesspoint



Bal4cCloudHowitallfitstogether

Bal<cGrid

Plannedac<vi<es

SA1‐3,NA1‐4,JRA

BGi‐Bal<cGridInnova<onLab

Bal<cCloud Courses SMEconnec<vity

BC



•  Bal<cGridaskedonecenterpercountrytoinstallacloudinstance

•  SNIC(SwedishNa<onalInfrastructureforCompu<ng)supportedthecoordina<onofthework

•  Learningbydoing‐what'sinitforeScience,really?Isindustryinterested?

•  Startedinforma<ongatheringandsharingthroughoursite;wenttocloudexposandconferences

•  Tookearlycontactwithopensourcecloudvendors,decidedtofocusonEucalyptus



Prac4cali4es&Learningbydoing

Firstusers

Firstusage:1.  Virtualworldinstalla<ons(Wonderland),for

theImmersiveEduca<onorganiza<on

2.  Filmrendering,forEstonianfilmcompany

Virtualworldsandmovies

July17,2009 94thTerenaTFStorageMee<ng,


Opensource“cloudinabox”

Our (main) choice:

-  Lead by Rich Wolski, UCSB. -  Integrated with -  Integrated with

10

BC


•  Willtheopensourcealterna<vesbesupportedinasa<sfactoryway?–  “Notinventedhere”easilybecomes“Notsupportedhere”

•  Earlysuccessopensourcealterna<vesbranchingoffaprivateversion–thinkXen,Eucalyptus,…–whichbranchwillgetthebestsupport?

•  Missingstandardsforinteroperability,cfgrids.

•  Lock‐inanddefactolock‐in,thebidding‐on‐a‐dollarsyndrome

MajorConcerns



Status–Bal4cCloud

4thTerenaTFStorageMee<ng,Copenhagen,2009.Sept.15,Bal<cCloud 12

SeehGp://cloud.bal4cgrid.eu

Status–Estonia‐NICPB(1/3)

KVM‐basedsetupwithEucalyptusinterface

Evalua<ng:•  othervirtualisa<ontechnologies,e.g.vmwaresolu<on

•  crea<onofthevirtualprivatenetworksusingEucalyptus,OpenVPNanddynamicDNS‐tobeabletoprovidesomethingliketherecentlyrolledoutAmazonVirtualPrivateCloud(h\p://aws.amazon.com/vpc/)

Usage:•  primarilyNICPB

•  ImmersiveEduca<onprojectwiththedevelopementanddemoresourcesforrunningtheProjectWonderland‐basedvirtualworlds




TheKVMinfrastructureisalsousedasapla_ormfortheongoing

SMEcollabora4on‐on‐demandrenderfarmcrea4on.TheplansaretomakeitcontrollablebyEucalyptus,butsofaritisnot

capableofsuppor<ngWindowsmachinesoutofthebox(ongoing).

Inves4ga4ngthecrea4onoftheprivatevirtualnetworks(usingOpenVPNanddynamicDNSsolu<ons)tobeabletoprovide

somethingliketherecentlyrolledoutAmazon

VirtualPrivateCloud(h\p://aws.amazon.com/vpc/).




Storage:

–  testbedinstalla<onofthehadoopFSinNICPBwithSRMinterfaces.TryingtomigratethestorageinfrastructurefromdCachetoHDFS.

–  Wins:easieradministra<on,cloudflavor‐canrunhadoopjobsmuchmoreefficiently.

Problems:–  Eucalyptusswiss<lltooalpha/pre‐beta.The1.6_should_bebe\er,

incl.supportformul<pledatacenters(availabilityzones)andmorerobustopera<ons,butit'snotoutyet.

–  Accoun<ngofconsumedresourcesiscomplicated–thevirtualisa<ontoolsarenotmakinganaccentonthat.



Status‐Latvia‐RTU

Installa4on:Eucalyptusinstalla<on,xen,4cpuforusers.HWupdatesexpected(userrequest).

Usage:Submi\edsimpletests,includingstudentstes<ngcloud

Issues:Systemnotquitestable.“imhosofarnotusableforseriousproject”.



Status‐Belarus–UIIPNASB(1/3)

Installa4on:

–  HW:6CPU,24cores(enoughforpresenttasks);

–  Virtualisa<on:Xen;–  Cloudtechnolgies:Eucalyptus,Walrus;–  Howtoreach:h\ps://80.94.168.123:8443/–  Status:3GNU/Linuximagesavailable,commonlyrunningfrom2to10

instances.

Usage:

‐  verifiedbyregularmanualmonitoringofhostsandservicesrunningwithinthecloud;

‐  usedbyUIIPNASBgridteamfordesigningdynamicgridresourcereserva<onandadjustmentmechanismsforUNICOREsites




Usage:pilotprojectofmanyusers

‐  wehavepreparedapilotprojectproposalondeployingoneoftheadvancedinforma<onsystemsoftheStateCommi\eeforScienceandTechnologiesoftheRepublicofBelarus(SCST)intothecloudinfrastructureprovidedbythedatacenteroftheUIIPNASB.

‐  Theinforma<onsystemtobecloudifiediscreatedandadministeredonbehalfoftheSCSTbyRepublicanCentreforTechnologyTransfer(h\p://ic\.by).Thesystemhasthousandsofvisitorsandhundredsofac<veuserswhoregularlypostandupdatetheirTechnologyOffersandRequests,disseminatenews,uploadtechnologytransferrelatedapplica<ons,graphicalandvideodata.

‐  Theprimarygoalofthepilotprojectisprovingtheconceptofcloudcompu<ngbeingagoodwaytogonotonlyforSMEsbutforsomegovernmentalagenciesaswell.




Plans:

UIIPNASBhaveplansforcloudusagegrowth,expansion,connec<ontothe

outsideworldandtoothercountries.[e.g.rightnowwearebuyinghardware

fortheexpansionofourcloudinfrastructure]

Issues:

ToincreasecloudinfrastructurestabilityweusecustomkernelbuildsforXenvirtualmachineimagesdeployedinthecloud.



Status‐Lithuania–VGTU(1/7)EucalyptusCloudScheme



  Clusterinstalla<onisdoneusingRocksClusters5.2(Chimichanga)

  VGTUcloudinstanceisavailableath\ps://euca.cloud.vgtu.lt:8443

  Eucalyptusv1.5.2isinstalled

  XENhypervisorisusedforparavirtualiza<on

  MANAGEDnetworkmodeisconfiguredi.e.machineinstancescanbeaccesseddirectlyfrominternet

  Availabletwovirtualmachineimagescompiledmanually:

  CentOS5.3(Basesetup)

  SLC53(Scien<ficLinuxCern)(Basesetup)

Status‐Lithuania–VGTU(2/7)Cloudclustersetup



Frontend(Clustercontroler,Cloudcontroler):

  1xPen<um4CPU,2GBRAM,200GBHDD,1Gbps

Nodes(Nodecontroler)

  4xPen<um4CPU,2GBRAM,200GBHDD,1Gbps

  2xQuadQoreCPU,4GBRAM,320GBHDD,1Gbps

AvailabilityzoneDEBESELIS

AVAILABILITYZONE DEBESELIS UPeuca.cloud.vgtu.lt

  AVAILABILITYZONE |‐vmtypes free/maxcpuramdisk

  AVAILABILITYZONE |‐m1.small 0016/001611283

  AVAILABILITYZONE |‐c1.medium 0016/001612565

  AVAILABILITYZONE |‐m1.large 0016/001615125

  AVAILABILITYZONE |‐m1.xlarge 0016/001617685

  AVAILABILITYZONE |‐c1.xlarge 0008/0008210245

Status‐Lithuania–VGTU(3/7)AvailableResources

Total12Cores



Status‐Lithuania–VGTU(4/7)TechnicalISSUES(problemsandlimita<ons)

  Inrecenteucalyptusversion1.5.2famouserror:

“Error500messageexpired”

thatpreventedgraphicaltoolslikeElas<cfox,Rightscale,Ylas<c,EC2Dreamfromworkingwitheucalyptusinstance,wasnotfixed

  Wehavefixedthisissuemanually:

Addedthiscodeto:

 HMACQuerySecurityHandler.javaasatemporaryfix‐:

tsFormat.setTimeZone(TimeZone.getTimeZone("EEST"));//AddthisLineandimportTimeZoneclass

ts.setTime(tsFormat.parse(<mestamp));

AndthenEucalyptusrpmswererebuilt.23


Status‐Lithuania–VGTU(5/7)WehaveinstalledanduseElas<cFoxforEucalyptusmanagement.



WehavetriedEC2DreamforEucalyptusmanagementaswell.



Status‐Lithuania–VGTU(6/7)

“Cloud computing” activity is included into LitGrid programme 2009.

Walrus based storage will be installed.

Status‐Sweden–PDCInstalla<on(ongoing):4quadcoremachines,Eucalyptus,XEN

Usage:fortes<ngandlearning,topreparelargerinstalla<onbasedonolderHPCresources



Status‐Lithuania–VGTU(7/7)Nearestfuture

Status‐Lithuania‐VU

Installa4on:Smalleucaluptusintalla<on(mightbeincreasedifdemandarrises),xenbased,hardwareisonly32bit.DoingsometestswithvmwareandMShyper‐v

Usage:Smalltests,nothinginproduc<on

Plans:Thinkingofexpanding(havetothinkmoreaboutwhichtechnology),maybesomeservicestosomeotherins<tu<ons

Issues:Currentsystemisnotverystable,someaccidentalcrashesandsystemhangingswereno<ced.



Status‐Poland–PSNCInstalla4on:

–  twodualCPUmachines(oneforcloudcontrolleroneforcloudnodes),2xdualcoreIntelXeon,8GMram/each

–  147GBSAShdd–  two1GbitEthnetworkcard/eachvirtualiza<on:kvmorxen(ondemand)–  cloudstack:eucalyptus1.5.2–  url:h\ps://bgcloud.man.poznan.pl:8443,currentlyavailableonlyfromPSNC

network

Usage:teststage

Plansforusageandexpansion:

–  aspartofBal<cCloud–  forlocalusers(PSNCdevelopersandresearchteam),educa<onins<tu<onin

Poznan

–  expansionaccordingtoloadandu<liza<onwecanprovideaddi<onalcloudnodecontrollers



•  VGTU:Bachelorthesisoncloudcompu<ng(2009):“DeploymentofServiceOrientedITInfrastructure”

•  VGTU:Cloudcompu<ngispartoftheMSccourse“Highperformancesystems”atVGTU

•  BG:generalpresenta<ons,e.g.atISSGC09•  NICPB:AcourseinITcollege(inTallinn)aboutthedistributedsystems.Cloudcompu<ngispartofthecourse,labswillberunontheBCresources

•  NICPB:Tartucloudandstartupschool–May,2009



Educa4on

•  Connec<ngtheBal<cCloudwiththeNorthernEuropeCloud‐Sweden,Norway,Denmark,Estonia,Latvia,Lithuania,Belarus,Poland+Iceland(?)+…

•  Availabilityzones–connec<ngtheclouds

•  Connec<ngwithotherclouds,e.g.intheNetherlands

•  Seemorecloudcompu<ngbeingusedbystartupsandin‐housequickprototyping

•  Seemorecloudcompu<ngbeingusedinacademia–esp.HPCflavours•  Seemoreopensourcealterna<ves,andalterna<vecloudproviders–alsoin

academia•  Improvedsolu<onsaddressingthedangersincloudcompu<ng.•  Lowexpecta<onsonstandardAPIsJuly17,2009 30


Future

Expecta4onson2009

•  Bal<cCloudhavelearnedquitemuchaboutthecurrentopensourcecloudtechnologies–whatthereistouse,whatismissing,thequalitylevel

•  Theusersideisevolving,nowthatwehavesomebasiccoursesandancloudstoplaywith

•  Nextlevel,beyondthispilotproject,istobepartoftheNorthernEuropeCloud

4thTerenaTFStorageMee<ng,Copenhagen,2009.Sept.15,Bal<cCloud 31

Cloudcompu4ng,Virtualisa4on&theFuture

Summary

?32


Thankyou!

AakeEdlund

[email protected]

www.pdc.kth.se/Members/edlund

www.bal<cgrid.eu(Bal<cGrid)

cloud.bal<cgrid.eu(Bal<cCloud)

bgin.wordpress.com(BGInnova<onLab)



Basics

CloudInterface

CloudController

ClusterController

NodeController



Cloudcontroller•  Webservice

–  Java•  Handlesdistribu<onofVMinstancecontroleventstoclustersof

resources

Cloudcontroller

SLAEngine

InstanceScheduler

OtherSchedulers?



CloudInterface

CloudController

ClusterController

NodeController

Clustercontroller•  Webservice

–  Cbased•  Handlesdistribu<onofVMinstancecontroleventstoindividualofresources•  Inves<ga<on:whataretheperformance,invasivenessandfunc<onalitytradeoffs

ofvariouscloudnetworkingstrategies?

Clustercontroller

VirtualNetwork

InstanceScheduler

OtherNetwork/Schedulersolu<ons?



CloudInterface

CloudController

ClusterController

NodeController

Cloudinterface•  Webservicebased

–  Wri\eninJava•  HandlesusereventtocloudVMinstancecontroltransla<on•  Experiment:instancecrea<on<me

Cloudinterface

EC2compa<bleuserinterface

Web‐basedadmininterface

Otherinterfaces?



CloudInterface

CloudController

ClusterController

NodeController

balc cloud - terena · outside world and to other countries. [e.g. right now we are buying hardware...

Documents