aws and scientific computing

!

This!Amazon!Web!Services,!Inc.!(AWS)!package!is!provided!for!informational!purposes!only.!The!services!included!in!this!package!are!standard!commercial!services.!This!package!may!include!a!set!of!suggested!solutions!for!this!opportunity!that!are!based!on!our!limited!information,!and!should!not!be!construed!as!a!binding!offer!from!AWS.!For!

current!prices!for!AWS!services,!please!refer!to!the!AWS!website!at!www.aws.amazon.com.!

!!

!!Amazon!Web!Services!and!!High5Performance!Computing!(HPC)!

!AWS!and!HighDPerformance!Computing!(HPC)! ! !

11!Aug!2015! ! Page!1!!

Copyright!©!2015!AMAZON!WEB!SERVICES,!INC.!!

!!20155Aug511!!!!!Prepared!by:!!Brendan!Bouffler!<[email protected]>!!Amazon!Web!Services,!Inc.!410!Terry!Avenue!North!Seattle,!WA!98109D5210!!


11!Aug!2015! ! Page!2!!


!!!!!!!

Why!AWS!for!High5Performance!Computing!(HPC)?!When!Amazon!Web!Services,!Inc.!(AWS)!emerged!from!the!Amazon.com!retail!business!in!2006,!the!scale!of!our!data!centers!already!rivaled!that!of!all!but!the!more!ambitious!TOP!500!sites!around!the!world,!allowing!an!efficient!exploitation!of!scale!economics!that!the!company!had!been!enjoying!for!some!time.!This!can!now!be!extended!to!our!customers!directly!through!what!was!then!a!unique!offering!of!“rentable!cycles.”!

The!virtualization!technology!used!allowed!the!provisioning!of!new!servers!(and!subsequently!new!services)!in!minutes,!compared!to!what!previously!took!weeks,!meaning!that!experimenting!with!new!services!and!ideas!became!a!lowDrisk,!highDreward!opportunity!and!thus!new!ideas!faced!a!low!barrier!to!entry.!It!also!meant!that!successful!ideas!could!be!permitted!to!scale!more!quickly.!This!flexibility!is!one!of!the!immediate!reasons!why!cloud!computing—with!AWS!as!its!pioneer—has!seen!broad!

success!in!the!enterprise!space!and!government!sector.!

The!other!key!factor!supporting!the!cloud’s!success!stems!from!a!simply!stated!maxim!that!you!“only+pay+for+what+you+use.”!The!obverse—also!true—is!much!more!compelling:!“you+don’t+pay+for+what+you+don’t+use.”!This!represents!a!new!economic!model!that!is!already!driving!the!next!era!in!scientific!computing:!the!ability!to!procure!specialist!system!components,!in!almost!any!quantity,!for!a!finite!period!of!time!and!to!discard!them!when!the!task!is!completed.!

This!payment!model!is!important!not!only!because!of!the!potential!to!save!costs!for!lightly!loaded!systems!(and!in!smallD!to!mediumDsized!deployments,!there!are!a!lot!of!those)!but!also!for!the!variety!it!offers!and!the!chance!for!experimentation.!A!researcher!can!form!a!hypothesis!that!a!particular!problem!may!be!tractable!as!a!mapDreduce!type!workload!(for!example)!and!quickly!build!and!run!the!model!to!test!it!without!ever!having!to!own!a!mapDreduce!cluster.!If!the!hypothesis!fails!the!test,!the!researcher!can!dispose!of!the!mapDreduce!facility!and!move!on,!without!penalty.!In!enterprise!parlance,!this!is!

Figure!1!–!Millions!of!uncoupled,!uncorrelated!workloads!combine!to!fill!up!the!cloud.!


11!Aug!2015! ! Page!3!!


called!“failing!fast,”!but!the!process!of!testing!ideas!and!allowing!them!to!fail!is!fundamental!to!science.+!

The!realization!of!not!paying!for!what!you!don’t!use!is!only!really!possible!at!scale!and!is!definitely!difficult!to!selfDadminister.!The!reason,!of!course,!is!that!unused!infrastructure!still!needs!to!be!paid!for!by!someone!and!having!sufficient!capacity!of!multiple!different!shapes,!sizes,!and!strengths!available!at!any!one!time!for!adDhoc!availability!means!that!when!it’s!not!being!used!by!one!particular!user,!the!facility!owner!has!sufficient!confidence!that!another!user!will!need!or!want!it.!To!have!that!level!of!confidence!without!exact!planning!or!direct!insight!into!users’!minds,!it’s!best!to!have!a!large!number!of!users!with!multiple,!uncorrelated!workloads.!In!AWS’s!case,!we!have!more!than!one!million!such!customers—definitely!sufficient!scale!for!making!statistical!inferences!work.!In!many!ways,!the!cloud!can!really!be!seen!as!a!crowdDsourcing!mechanism!for!sharing!infrastructure.!

Another!benefit!of!this!ownership!model!(or,!rather,!this!lack5of5ownership!model)!is!that!some!workloads!can!be!sped!up!dramatically.!An!embarrassingly!parallel!workload!is!characterized!by!a!lack!of!interaction!or!strong!dependency!between!the!multiple!(sometimes!thousands)!of!arbitrary!threads.!Great!examples!of!this!abound!in!the!sciences—free!energy!perturbation!in!chemistry,!ensemble!Fourier!transforms,!whole!classes!of!algorithms!in!bioinformatics,!and!Monte!Carlo!methods!all!share!this!characteristic.!This!class!of!problems!commonly!shares!the!property!that!a!single!core!laboring!for!hundreds!of!hours!will!produce!the!same!output!as!hundreds!of!cores!laboring!for!one!hour.!In!a!cloud!model!like!that!described!here,!the!cost!to!the!end!user!will!be!the!same,!but!in!one!of!these!cases,!the!workload!returns!its!result!after!an!hour!and!the!other!nearly!a!week.!

The!Spot!Market!Depth—the!absolute!scale!of!resources—is!the!final!element!to!ensuring!that!highly!scalable!workloads!like!these!are!able!to!benefit.!Think!of!it!like!the!sheer!number!of!cores!available!for!compute,!for!example.!Ideally,!a!user!with!a!characteristic!need!like!that!discussed!above!would!like!the!ability!to!reach!out!and!find!an!almost!unlimited!pool!of!resources!in!order!to!scale!their!workload!up!(i.e.,!shorten!their!effective!waiting!

Figure!2!–!Trading!time!for!capacity:!In!the!case!of!embarrassingly!parallel!workloads,!speed!increases!can!be!

dramatic!and!virtually!free.!


11!Aug!2015! ! Page!4!!


time)!by!many!multiples.!In!the!first!instance,!the!scale!of!resources!demanded!by!more!than!a!million!customers!provides!a!lot!of!depth.1!

But!to!cope!with!some!of!those!users!suddenly!and!dramatically!departing!from!their!usual!usage!profile,!we!need!to!provide!sufficient!headroom!or!reserve!capacity!and!this,!as!we!know,!can!present!a!risk,!since!that!excess!capacity!can!effectively!cost!every!user!of!the!cloud!if!it’s!allowed!to!go!unused.!For!this!reason!(and!many!others),!AWS!operates!a!Spot!market!for!unused!compute!cores.!Similar!to!any!market,!the!buyers!place!bids!for!resources!that!are!in!varying!states!of!supply!(depending!on!demand!and!their!characteristics)!and!so!effectively!participate!in!the!price!decisions!being!made!minute!to!minute!in!the!cloud.!This!means!that!a!buyer!can!be!found!for!most!resources!available!in!the!cloud!almost!regardless!of!the!state!of!the!market.!It!will!just!be!the!price!that!varies.!For!example,!GPUDenabled!resources!might!become!especially!cheap!(“in!excess!supply”)!when!a!large,!VDIDbased!call!center!closes!for!business.!The!skills!needed!to!adapt!to!the!rigors!of!a!floating!market!like!this!can!be!trivial!for!some!and!hard!for!others.!However,!with!a!large!enough!user!base,!sufficient!numbers!of!users!will!be!present!to!develop!the!techniques!necessary!to!exploit!the!Spot!market!and!thus!find!it!interesting!in!which!to!participate.!

The!scientific!community!has!been!a!very!large!and!enthusiastic!user!of!the!Spot!market,!since!scientific!computing!algorithms!span!the!length!and!breadth!of!workload!possibilities.!It!is!for!these!reasons!that!scientific!computing!is!such!a!complementary!workload!for!the!cloud.!Moreover,!for!workloads!that!are!not!timeDsensitive,!the!costs!can!be!extraordinarily!low.!

Tools!for!Provisioning!HPC!Clusters!in!the!Cloud!AWS!is!built!upon!a!broad!and!deep!core!set!of!cloud!infrastructure!services!that!run!the!full!spectrum!of!requirements!for!compute!workloads.!

In!the!simplest!case,!elastic!compute,!network,!and!storage!services!can!be!combined!in!a!limitless!variety!of!ways!to!create!very!complex,!resilient,!and!powerful!solutions.!The!most!common!HPC!workloads!typically!make!use!of!some!very!common!components!to!build!the!basic!HPC!structure!and!then!expand!in!interesting!new!ways!to!exploit!new!AWS!cloud!services!as!they!are!released.!AWS’s!goal!is!to!constantly!free!you!up!to!do!great!science,!rather!than!the!undifferentiated!heavy!lifting!of!building!and!running!a!cluster!of!IT!services.!

The!AWS!cfnCluster!tool!(discussed!in!more!detail!below),!for!example,!makes!use!of:!

•! Amazon!Elastic!Compute!Cloud!(Amazon!EC2)!compute!services!to!provide:!o! Hosts!running!CentOS!Linux!with!highDperformance!Intel!Xeon!(Ivy!Bridge!and!Haswell)!cores.!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!1!See!James!Hamilton,!our!VP!and!Distinguished!Engineer,!talking!about!the!scale!of!our!infrastructure!here:!https://www.youtube.com/watch?v=JIQETrFC_SQ.!


11!Aug!2015! ! Page!5!!


o! Auto!Scaling!Groups!to!allow!for!expanding!the!number!of!compute!node!instances!when!the!cluster!is!under!load.!

•! Amazon!Elastic!Block!Store!(Amazon!EBS)!to!provide!highDperformance!block!devices!that!are!then!formatted!with!the!ext4!file!system!and!ready!for!local!scratch!or!shared!file!system!uses.!

•! Enhanced!networking!and!Virtual!Private!Cloud!(VPC)!to!provide!highDperformance!10!Gb/s!networking!in!an!incredibly!secure!environment!to!ensure!complete!customer!privacy!and!segregation.!

•! AWS!CloudFormation,!an!AWS!cloud!service!built!around!customizable!templates!that!gives!developers!and!systems!administrators!an!easy!way!to!create!and!manage!a!collection!of!related!AWS!resources,!provisioning!and!updating!them!in!an!orderly!and!predictable!fashion.!

There!are!many!dozens!of!services!offered!by!AWS,!as!shown!in!the!diagram!below.!They!range!from!straightforward!infrastructure!services!to!complex!offerings!like!Amazon!Elastic!MapReduce!(Amazon!EMR)!or!highDperformance!message!services!like!Amazon!Kinesis.!New!ideas!and!new!services!are!created!every!day!by!more!than!a!million!customers!using!these!building!blocks.!Our!continual!innovation!ensures!that!customers!maintain!stateDofDtheDart!IT!infrastructure,!enhancing!performance!and!versatility.!AWS!has!launched!a!total!of!1,515!new!features!and/or!services!since!inception!in!2006.!

!

Figure!3!–!High5Level!Overview!of!AWS!Cloud!Services!

More!Information!The!complete!set!of!AWS!cloud!services!is!too!large!to!simply!outline!here!and!is!constantly!expanding.!The!most!upDtoDdate!reference!for!available!services!is!the!cloud!itself.!There!is!a!lot!of!good!information!available!at!aws.amazon.com,!in!our!


11!Aug!2015! ! Page!6!!


documentation!at!aws.amazon.com/documentation,!and!in!the!whitepapers!we’ve!written!at!aws.amazon.com/whitepapers.!

Multiple!Approaches!to!HPC!in!AWS!The!sections!below!outline!several!approaches!to!HPC!in!AWS.!

cfnCluster!cfnCluster!is!AWS’s!own!set!of!openDsource!and!freely!available!tools!for!quickly!provisioning!an!HPC!cluster!in!the!cloud.!In!true!openDsource!spirit,!it’s!also!freely!modifiable!to!suit!your!own!needs.!

Based!on!Python,!and!using!key!AWS!cloud!services!such!as!Amazon!EC2,!Amazon!EBS,!and!AWS!CloudFormation,!cfnCluster!allows!for!easy!creation!of!traditional!multiDnode!HPC!clusters!using!CentOS!and!a!selection!of!standard!MPI!libraries!and!common!schedulers.!

These!clusters!are!elastic!in!that!they!can!dynamically!expand!(up!to!your!preconfigured!limit)!to!service!a!growing!queue!of!jobs!and!shrink!(when!the!queue!is!depleted).!A!single!site!can!have!any!arbitrary!number!of!clusters!operating,!some!or!all!of!which!can!be!ephemeral—simply!ceasing!to!exist!when!they!are!no!longer!needed.!Individual!clusters!can!share!common!file!systems!or!be!completely!different,!allowing!for!some!software!stacks!to!be!“freezeDdried”!in!the!cloud!and!rehydrated!at!a!later!date!should!it!be!necessary!to!recreate!a!workload!or!revive!a!calculation!method!for!further!work.!

cfnCluster!is!able!to!take!full!advantage!of!the!AWS!Spot!market,!making!it!easy!to!consume!thousands!of!potentially!inexpensive!cores!to!accelerate!progress!in!the!quiet!hours!of!the!night!when!the!rest!of!your!selected!region’s!workloads!are!sleeping—a!unique!proposition!that!science!is!truly!able!to!take!advantage!of!in!many!cases.!

Bright!Cluster!Manager!Bright’s!Cluster!Manager!comes!to!the!cloud!from!a!long!history!of!managing!onDpremises!traditional!HPC!clusters.!

Bright!enables!the!creation!of!a!complete!cluster!in!Amazon!EC2!or!easily!extends!your!onsite!HPC!cluster!into!the!cloud.!Bright’s!Cluster!Manager!provides!a!“single!pane!of!glass”!to!both!your!onDpremises!and!cloud!resources,!enabling!you!to!dynamically!add!capacity!and!manage!cloud!nodes!as!part!of!your!onDsite!cluster.!Cloud!utilization!can!be!achieved!in!just!a!few!mouse!clicks,!without!the!need!for!expert!knowledge!of!Linux!or!cloud!computing.!Queues!can!be!created!and!resources!directed!to!the!cloud,!allowing!an!efficient!control!of!resource!provisioning!and!utilization.!

A!great!advantage!of!Bright!is!that!the!software!environment!in!the!cloud!instances!is!completely!identical!to!that!which!is!provisioned!onDpremises—meaning!less!software!complexity!for!users!to!deal!with!and!fewer!points!of!management!for!the!maintainers!of!the!clusters.!Bright!is!also!able!to!take!advantage!of!the!AWS!Spot!market.!


11!Aug!2015! ! Page!7!!


While!it’s!still!a!task!for!an!HPC!facility!manager!to!decide!what!workloads!should!be!enabled!for!cloud!extension!and!how/when!or!if!their!data!should!be!preDprovisioned!for!them!in!the!cloud!(since!these!aspects!of!the!workflow!are!highly!specific!to!a!workgroup),!Bright!Cluster!Manager!facilitates!the!provisioning!and!management!step,!allowing!for!a!much!easier!approach!to!managing!these!tasks.!

Rescale!Rescale!operates!an!awardDwinning!cloud!simulation!and!HPC!platform!and!can!be!deployed!within!minutes!to!any!organization.!

The!ScaleX!Pro™!platform!is!designed!for!independent!professionals!and!businesses!to!perform!their!complex!engineering!and!scientific!simulations!with!speed,!performance,!and!confidence.!

ScaleX!Developer™!is!designed!for!external!application!developers!and!Independent!Software!Vendors!(ISVs)!to!build,!test,!and!deploy!software!directly!to!Rescale’s!platforms!and!perform!native!software!integration!with!Rescale’s!powerful!backend.!

ScaleX™!Enterprise!is!the!enterprise!deployment!of!Rescale’s!industryDleading!cloud!simulation!and!HPC!platform.!ScaleX!Enterprise!features!a!unified!enterprise!simulation!platform!and!a!powerful!administrative!portal,!along!with!direct!integrations!and!management!of!onDpremises!HPC!resources,!schedulers,!and!software!licenses.!

Cycle!Computing!Cycle!Computing!(Cycle)!migrates!and!orchestrates!big!data!and!big!compute!workloads!across!private!and!public!cluster!and!cloud!resources.!Cycle!has!experience!helping!customers!leverage!the!cloud!for!a!wide!variety!of!applications,!including:!!

•! OpenDsource!and!commercial!genomic!and!drug!discovery!applications!for!life!sciences.!

•! Commercial!CFD!and!CAD!software!for!manufacturing.!•! Monte!Carlo!and!risk!analysis!applications!for!financial!services.!•! Internally!created!code!for!enterprise!or!government.!

!Cycle!operates!an!AWS!test!drive!facility!that!allows!our!customers!to!gain!handsDon!experience!with!real!workloads!in!order!to!explore!the!possibilities!of!what!Cycle’s!platform!can!offer.!

Conclusion!The!cloud!truly!has!become!the!new!normal!in!science!and!is!delivering!huge!benefits!due!to!its!inherent!flexibility!in!conducting!experiments!and!trials!with!what!was!previously!expensive!and!difficultDtoDmanage!infrastructure.!It!permits!access!to!resources!at!a!scale!that!most!researchers!would!rarely!be!familiar!with!and!promotes!a!mechanism!for!sharing!data!and!workloads!that!leaves!everyone!free!to!focus!on!their!science.!


11!Aug!2015! ! Page!8!!


!AWS!has!solution!architects!around!the!globe!who!are!very!interested!in!working!with!the!research!computing!community!to!better!understand!what!they!can!build!in!AWS.!!AWS!also!offers!a!scientific!computing!organization!that!works!every!day!to!make!the!cloud!more!and!more!useful!to!scientists!and!researchers!everywhere.!!

aws and scientific computing

Technology