HPC Clusters in the [almost]* Infinite cloud

Global Scientific Computing

2016-04-21

*Does not apply to mathematicians with specialties in Cantorian set theory who should immediately ask for a copy of my very long disclaimer.

We are Psycho SciCo

Science is one of the greatest areas ofcomputation

Amazonhuge, really disruptive, impact

what we are about

Team

The Scientific Computing (SciCo) team is a global group of scientists and specialists from Amazon Web Services.

We’re responsible for making the sure the cloud continually innovates in ways that benefit the global community of researchers from whom we draw our inspiration.

Our aim is to bring the revolutionary benefits of agility and extreme scale to this community so we can all keep making the discoveries that will change the world and impact the lives of everyone on our planet.

We have team members from physics, astronomy, aeronautical engineering, and genomics and all have extensive experience in research and high performance computing. We even have a rocket scientist.

Peak: 58K cores

Valley: 12K cores

“… the online book and decorative pillow sellerAmazon.com swooped in and, in 2006, launched its own computer rental system—the future Amazon Web Services. The once-fledgling service has since turned cloud computing into a mainstream phenomenon …”

Source: Bloomberg Business - April 22, 2015

$7B retail business10,000 employees

A whole lot of servers

2006 2015

Every day, AWS adds enough server capacity to power this $7B enterprise

Existing1. Oregon2. California3. Virginia4. Dublin5. Frankfurt6. Singapore7. Sydney8. Seoul9. Tokyo10. Sao Paulo11. Beijng12. US GovCloud

1. Ohio2. India3. UK4. Canada5. China+1

AWS Region Availability Zone

regions are sovereign your data never leaves

Map of scientific collaboration between researchers - Olivier H. Beauchesne - http://bit.ly/e9ekP2

Science means Collaboration

More time spent computing the data than moving the data.

Cray Supercomputer

Beowulf Cluster

A top500 supercomputer

Disrupting science, wherever it’s happening.

9600 Strips (~80TB) to be delivered to GSFC

~1600 strips (~20TB) at GSFC

Area Of Interest (AOI) for Sub-Saharan Arid and Semi-arid Africa

core

s

time

core

s

time

Meeeeelions

core

s

time

Collectiveaction

Wheneveryone comestogetherinthecloud tosharetheresource, andonlypaysforwhattheyuse,theefficiency ishuge.

Spot Market

core

s

time

Spot Market

Our ultimate space filler.

Spot Instances allow you to name your own price for spare AWS EC2 computing capacity.

Great for workloads that aren’t time sensitive, and especially popular in research (hint: it’s really cheap).

Pricing Models

Cost Control & Budgeting

Spot Bid Advisor

The Spot Bid Advisor analyzes Spot price history to help you determine a bid price that suits your needs.

You should weigh your application’s tolerance for interruption and your cost saving goals when selecting a Spot instance and bid price.

The lower your frequency of being outbid, the longer your Spot instances are likely to run without interruption.

https://aws.amazon.com/ec2/spot/bid-advisor/

Bid Price & Savings

Your bid price affects your ranking when it comes to acquiring resources in the SPOT market, and is the maximum price you will pay.

But frequently you’ll pay a lot less.

https://aws.amazon.com/ec2/spot/bid-advisor/

Spot Bid Advisor

As usual with AWS, anything you can do with the web console, you can do with an API or command line.

bash-3.2$ python get_spot_duration.py \--region us-east-1 \--product-description 'Linux/UNIX' \--bids c3.large:0.05,c3.xlarge:0.105,c3.2xlarge:0.21,c3.4xlarge:0.42,c3.8xlarge:0.84

Duration Instance Type Availability Zone168.0 c3.8xlarge us-east-1a168.0 c3.8xlarge us-east-1d168.0 c3.8xlarge us-east-1e168.0 c3.4xlarge us-east-1b168.0 c3.4xlarge us-east-1d168.0 c3.4xlarge us-east-1e168.0 c3.xlarge us-east-1d168.0 c3.xlarge us-east-1e168.0 c3.large us-east-1b168.0 c3.large us-east-1d168.0 c3.large us-east-1e168.0 c3.2xlarge us-east-1b168.0 c3.2xlarge us-east-1e117.7 c3.large us-east-1a36.1 c3.2xlarge us-east-1d34.5 c3.4xlarge us-east-1a23.0 c3.xlarge us-east-1b21.9 c3.2xlarge us-east-1a17.3 c3.8xlarge us-east-1b0.1 c3.xlarge us-east-1a

Peak: 58K cores

Valley: 12K cores

Wall clock time: ~1 hour Wall clock time: ~1 week

Cost: the same

When you only pay for what you use …

• If you’re only able to use your compute, say, 30% of the time, you only pay for that time.

1 Pocket the savings• Buy chocolate• Buy a spectrometer• Hire a scientist.

2 Go faster• Use 3x the cores to

run your jobs at 3x the speed.

3Go Large• Do 3x the science,

or consume 3x the data.

… you have options.

39 years of computational chemistry in 9 hoursNovartis ran a project that involved virtually screening 10 million compounds against a common cancer target in less than a week. They calculated that it would take 50,000 cores and close to a $40 million investment if they wanted to run the experiment internally.

Partnering with Cycle Computing and Amazon Web Services (AWS), Novartis built a platform thst ran across 10,600 Spot Instances (~87,000 cores) and allowed Novartis to conduct 39 years of computational chemistry in 9 hours for a cost of $4,232. Out of the 10 million compounds screened, three were successfully identified.

CHILES will produce the first HI deep field, to be carried out with the VLA in B array and covering a redshift range from z=0 to z=0.45. The field is centered at the COSMOS field. It will produce neutral hydrogen images of at least 300 galaxies spread over the entire redshift range.

The team at ICRAR in Australia have been able to implement the entire processing pipeline in the cloud for around $2,000 per month by exploiting the SPOT market, which means the $1.75M they otherwise needed to spend on an HPC cluster can be spent on way cooler things that impact their research … like astronomers.

SKA

https://aws.amazon.com/blogs/aws/new-astrocompute-in-the-cloud-grants-program/

AWS is providing a grants pool of AWS credits and up to one petabyte of storage for an AWS Public Data Set in order to support the development of an “Appstore for Astronomy”.

The data set will be initially provided by several of the SKA’s precursor telescopes including CSIRO’s ASKAP, ICRAR’s MWA in Australia, and KAT-7 (pathfinder to the SKA precursor telescope Meerkat) in South Africa.

The grants are open to anyone who is making use of radio astronomical telescopes or radio astronomical data resources around the world.

The grants will be administered by the SKA. They will be looking for innovating, cloud-based algorithms and tools that will be able to handle and process this never ending data stream. You can also read their post on Seeing Stars Through the Cloud.

not

http://blog.csiro.au/wtf-is-that-how-were-trawling-the-universe-for-the-unknown/

WTF’s cloud-based backend is hosted on Amazon Web Services servers, where the researchers are able to access software for data reduction, calibration and viewing right from their desktop. The team is currently issuing a challenge using data peppered with “EMU (Easter) Eggs” – objects that might pose a challenge to data mining algorithms.

This way they hope to train the system to recognise things that systematically depart from known categories of astronomical objects, to help better prepare for unanticipated discoveries that would otherwise remain hidden.

“The Zooniverse is heavily reliant on Amazon Web Services (AWS), particularly Elastic Compute Cloud (EC2) virtual private servers and Simple Storage Service (S3) data storage. AWS is the most cost-effective solution for the dynamic needs of Zooniverse’s infrastructure …”http://wwwconference.org/proceedings/www2014/companion/p1049.pdf

The World’s Largest Citizen Science Platform

… cost is a factor – running a central API means that when the Zooniverse is quiet and there aren’t many people about we can scale back the number of servers we’re running (automagically on Amazon Web Services) to a minimal level.

AWS Building blocks

TECHNICAL & BUSINESS SUPPORT

Account Management

Support

Professional Services

Solutions Architects

Training & Certification

Security & Pricing Reports

Partner Ecosystem

AWSMARKETPLACE

Backup

Big Data& HPC

Business Apps

Databases

Development

IndustrySolutions

Security

MANAGEMENTTOOLS

Queuing

Notifications

Search

Orchestration

Email

ENTERPRISEAPPS

VirtualDesktops

StorageGateway

Sharing &Collaboration

Email &Calendaring

Directories

HYBRID CLOUDMANAGEMENT

Backups

Deployment

DirectConnect

IdentityFederation

IntegratedManagement

SECURITY &MANAGEMENT

Virtual PrivateNetworks

Identity &Access

EncryptionKeys

Configuration Monitoring Dedicated

INFRASTRUCTURESERVICES

Regions AvailabilityZones Compute

StorageObjects, Blocks, Files

DatabasesSQL, NoSQL, Caching

CDNNetworking

PLATFORMSERVICES

App

Mobile & WebFront-end

Functions

Identity

Data Store

Real-time

Development

Containers

SourceCode

BuildTools

Deployment

DevOps

Mobile

Sync

Identity

PushNotifications

MobileAnalytics

MobileBackend

Analytics

DataWarehousing

Hadoop

Streaming

DataPipelines

MachineLearning

EC2

http://aws.amazon.com/ec2/instance-types/

There’s a couple dozen EC2 compute instance types alone, each of which is optimized for different things.

One size does not fit all.

C4Intel Xeon E5-2666 v3, custom built for AWS.

Intel Haswell, 16 FLOPS/tick

2.9 GHz, turbo to 3.5 GHz

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/c4-instances.html

Feature Specification

Processor Number E5-2666 v3

Intel® Smart Cache 25 MiB

Instruction Set 64-bit

Instruction Set Extensions AVX 2.0

Lithography 22 nm

Processor Base Frequency 2.9 GHz

Max All Core Turbo Frequency 3.2 GHz

Max Turbo Frequency 3.5 GHz (available on c4.2xLarge)

Intel® Turbo Boost Technology 2.0

Intel® vPro Technology Yes

Intel® Hyper-Threading Technology Yes

Intel® Virtualization Technology (VT-x) Yes

Intel® Virtualization Technology for Directed

I/O (VT-d)

Yes

Intel® VT-x with Extended Page Tables (EPT) Yes

Intel® 64 Yes

API

JavaPythonRubyPHPShell…

… and in most popular languages.

http://boto.readthedocs.org/en/latest/

Almost anything you can do in the GUI, you can do on the command line.

Anything you can do in the GUI, you can do on the command line.

as-create-launch-config spotlc-5cents--image-id ami-e565ba8c--instance-type m1.small--spot-price “0.05”

. . .as-create-auto-scaling-group spotasg

--launch-configuration spotlc-5cents--availability-zones “us-east-1a,us-east-1b”--max-size 16 --min-size 1 --desiredcapacity 3

cfnCluster - provision an HPC cluster in minutes

#cfnclusterhttps://github.com/awslabs/cfncluster

cfncluster is a sample code framework that deploys and maintains clusters on AWS. It is reasonably agnostic to what the cluster is for and can easily be extended to support different frameworks. The CLI is stateless, everything is done using CloudFormation or resources within AWS.

10 minutes

http://boofla.io/u/cfnCluster – (Boof’s HOWTO slides)

Headnode

Instance

Compute node

Instance

Compute node

Instance

Compute node

Instance

Compute node

Instance

10G Network

Auto-scaling groupVirtual Private Cloud

/shared

Head Instance• 2 or more cores (as needed)• CentOS 6.x, 7.x, Amazon Linux• OpenMPI, gcc etc…

Choice of scheduler:• Torque, SGE, OpenLava,

SLURM

Compute Instances• 2 or more cores (as needed)• CentOS 6.x, 7.x, Amazon Linux

Auto Scaling group driven by scheduler queue length.

Can start with 0 (zero) nodes and only scale when there are jobs.

ComputeNode

Instance

ComputeNode

Instance

ComputeNode

Instance

ComputeNode

Instance

ComputeNode

Instance

Clusters in the Cloud

Infrastructure as code

#cfncluster

The creation process might take a few minutes (maybe up to 5 mins or so, depending on how you configured it.

Because the API to Cloud Formation (the service that does all the orchestration) is asynchronous, we can kill the terminal session if we wanted to and watch the whole show from the AWS console (where you’ll find it all under the “Cloud Formation”dashboard in the events tab for this stack.

$ cfnCluster create boof-clusterStarting: boof-clusterStatus: cfncluster-boof-cluster - CREATE_COMPLETE Output:"MasterPrivateIP"="10.0.0.17"Output:"MasterPublicIP"="54.66.174.113"Output:"GangliaPrivateURL"="http://10.0.0.17/ganglia/"Output:"GangliaPublicURL"="http://54.66.174.113/ganglia/"

Yes, it’s a real HPC cluster

#cfncluster

Now you have a cluster, probably running CentOS 6.x, with Sun Grid Engine as a default scheduler, and openMPI and a bunch of other stuff installed. You also have a shared filesystem in /shared and an autoscaling group ready to expand the number of compute nodes in the cluster when the existing ones get busy.

You can customize quite a lot via the .cfncluster/config file - check out the comments.

arthur ~ [26] $ cfnCluster create boof-clusterStarting: boof-clusterStatus: cfncluster-boof-cluster - CREATE_COMPLETE Output:"MasterPrivateIP"="10.0.0.17"Output:"MasterPublicIP"="54.66.174.113"Output:"GangliaPrivateURL"="http://10.0.0.17/ganglia/"Output:"GangliaPublicURL"="http://54.66.174.113/ganglia/"

arthur ~ [27] $ ssh ec2-user@54.66.174.113The authenticity of host '54.66.174.113 (54.66.174.113)' can't be established.RSA key fingerprint is 45:3e:17:76:1d:01:13:d8:d4:40:1a:74:91:77:73:31.Are you sure you want to continue connecting (yes/no)? yesWarning: Permanently added '54.66.174.113' (RSA) to the list of known hosts.

[ec2-user@ip-10-0-0-17 ~]$ dfFilesystem 1K-blocks Used Available Use% Mounted on/dev/xvda1 10185764 7022736 2639040 73% /tmpfs 509312 0 509312 0% /dev/shm/dev/xvdf 20961280 32928 20928352 1% /shared

[ec2-user@ip-10-0-0-17 ~]$ qhostHOSTNAME ARCH NCPU NSOC NCOR NTHR LOAD MEMTOT MEMUSE SWAPTO SWAPUS----------------------------------------------------------------------------------------------global - - - - - - - - - -ip-10-0-0-136 lx-amd64 8 1 4 8 - 14.6G - 1024.0M -ip-10-0-0-154 lx-amd64 8 1 4 8 - 14.6G - 1024.0M -[ec2-user@ip-10-0-0-17 ~]$ qstat[ec2-user@ip-10-0-0-17 ~]$

[ec2-user@ip-10-0-0-17 ~]$ ed hw.qsubhw.qsub: No such file or directorya#!/bin/bash##$ -cwd#$ -j y#$ -pe mpi 2#$ -S /bin/bash#module load openmpi-x86_64mpirun -np 2 hostname.w110q[ec2-user@ip-10-0-0-17 ~]$ lltotal 4-rw-rw-r-- 1 ec2-user ec2-user 110 Feb 1 05:57 hw.qsub[ec2-user@ip-10-0-0-17 ~]$ qsub hw.qsub Your job 1 ("hw.qsub") has been submitted[ec2-user@ip-10-0-0-17 ~]$ [ec2-user@ip-10-0-0-17 ~]$ qstatjob-ID prior name user state submit/start at queue slots ja-task-ID ------------------------------------------------------------------------------------------------

1 0.55500 hw.qsub ec2-user r 02/01/2015 05:57:25 all.q@ip-10-0-0-44.ap-southeas 2 [ec2-user@ip-10-0-0-17 ~]$ qstat[ec2-user@ip-10-0-0-17 ~]$ ls -ltotal 8-rw-rw-r-- 1 ec2-user ec2-user 110 Feb 1 05:57 hw.qsub-rw-r--r-- 1 ec2-user ec2-user 26 Feb 1 05:57 hw.qsub.o1[ec2-user@ip-10-0-0-17 ~]$ cat hw.qsub.o1 ip-10-0-0-136ip-10-0-0-154[ec2-user@ip-10-0-0-17 ~]$

System-wide Upgrade from Ivy Bridge to Haswell

#cfncluster

$ ed ~/.cfncluster/config/compute_instance_type/compute_instance_type = c3.8xLarges/c3/c4/pcompute_instance_type = c4.8xLargew949$ cfncluster update boof-cluster

Downgrading is just as easy. Honest.

really

cfnCluster options to explore …

#cfncluster

# (defaults to t2.micro for default template)compute_instance_type = c4.4xLarge# Master Server EC2 instance type# (defaults to t2.micro for default templatemaster_instance_type = c4.4xLarge# Inital number of EC2 instances to launch as compute nodes in the cluster.# (defaults to 2 for default template)initial_queue_size = 0# Maximum number of EC2 instances that can be launched in the cluster.# (defaults to 10 for the default template)max_queue_size = 64# Boolean flag to set autoscaling group to maintain initial size and scale back# (defaults to false for the default template)maintain_initial_size = true# Cluster scheduler# (defaults to sge for the default template)scheduler = sge# Type of cluster to launch i.e. ondemand or spot# (defaults to ondemand for the default template)cluster_type = spot# Spot price for the ComputeFleetspot_price = 0.12

# Cluster placement group. This placement group must already exist.# (defaults to NONE for the default template)#placement_group = NONE

http://boofla.io/u/cfnCluster

Spending & Budgeting

#cfncluster

It’s easy to see very quickly how you’ve spent, and on what services …

Spending Control 2/2

#cfncluster

… and even easier to set limits and alerts, based on your budget.

The SciCo Team

Adrian White

AdrianWhiteleadsourScientificComputingbusinessacrossAsia.HejoinedAWSatit’sinception inAustraliain2012andspentseveralyearsasaSolutionsArchitect,helpinghundredsofcustomersbuildhighlyoptimizedsolutionsinthecloud,andteachingthousandsmorethroughhisworkshopsatourglobalsummitsacrossAsia.Adriancomesfromabackgrounddeep insoftwaredevelopmentaswellasapplicationandcontentdeliveryanddeployment.Thismeanshe'sveryhappy talkingaboutautomation,applicationdeploymentandsoftwaredevelopmentinthecloudwithalmostanyoneandcandemystifyalmostanything.HeholdsadoublemajorinComputerScienceandPhysicsfromtheUniversityofNewSouthWalesinAustralia,andhasneverreallyfullyrecovered.

BIO

Angel Pizarro

Angel Pizarro had 15 years of experience inbioinformatics (the computational side ofbiomedical research) prior to joining AWS in2013. Angel is the global lead SciCo for genomicsand life science research computing initiatives atAWS. The goal of the initiatives are to acceleratethe adoption of cloud computing in the globalcommunity of biomedical researchers.

Prior to AWS, Angel led the Bioinformatics andHPC resources at University of PennsylvaniaMedical School.

BIO

Anna Greenwood

Anna Greenwood joined AWS in 2015 to managethe AWS Cloud Credits for Research Program.Through this program, AWS provides credits toenable customers to pilot their researchworkloads on AWS and to develop tools thatadvance scientific research.

Prior to joining AWS, Anna was a researchscientist at Fred Hutchinson Cancer ResearchCenter where she led research projects related togenomics and neuroscience. In some circles sheis known by the prestigious title “The MacGyverof Fish Neuroscience”. Anna has a PhD inNeuroscience from Stanford University and a BSfrom Rutgers University.

BIO

Brendan Bouffler (“boof”)

Brendan Bouffler has 20 years of experience inthe global IT industry dealing with very largesystems in high performance environments. Hehas been responsible for designing and buildinghundreds of HPC systems for commercialenterprises as well as research and defencesectors all around the world and has quite anumber of his efforts listed in the top500,including some that have placed in the top 5.

Brendan previously lead the HPC Organization forDell in Asia and joined Amazon in 2014 to helpaccelerate the adoption of cloud computing inthe scientific community globally. He holds adegree in Physics and an interest in testingseveral of its laws as they apply to bicycles. Thishas frequently resulted in hospitalization.

BIO

Jeff Layton

Dr. Jeffrey Layton is an official rocket scientist andhas been kicking round in the HPC industry for morethan 25 years.Jeff has served in roles as a Professor, Engineer andScientist at Boeing, Lockheed Martin, NASA, andClarkson University, and has led technical efforts forHigh Performance Computing companies such asIntel, Panasas, and Dell.He’s been a cluster builder, user and code monkey, acluster admin, as well as a systems engineer,manager, and benchmark engineer for HPC vendors.He is also an active contributor to multiple opensource projects and writes for technical publications,magazines, books, and websites.His Ph.D. is from Purdue in Aeronautical andAstronautical Engineering.

BIO

Kevin Jorissen

Kevin Jorissen has 10 years of experience incomputational science. He developed softwaresolving the quantum physics equations for lightabsorption by materials, taught workshops toscientists worldwide, and wrote about highperformance computing in the cloud before itwas fashionable. He worked as a postdoctoralresearcher in Antwerp, Lausanne, Seattle, andZurich.

Kevin joined Amazon in 2015 to help acceleratethe adoption of cloud computing in the scientificcommunity globally. He holds a Ph.D. in Physics,but feels equally proud of growing parsnips,surviving a night bus from Darjeeling to Kolkata,and adapting to medium-spicy Sichuan food. Hethinks a five-hour run in the Cascade mountainsis a fineway to spend a free Saturday.

BIO

Michael Kokorowski

Dr. Michael Kokorowski is a space geek withhalf a decade spent at NASA’s Jet PropulsionLaboratory and another half a dozen years at theUniversity of Washington in Seattle studyingmagnetospheric physics.Michael is obsessed with getting the mostscience bang for the buck. He has experiencemanaging complex scientific projects, deliveringresults for NASA and NSF.He has experiments that have flown to Earth’sstratosphere, were lost on the Antarctic icesheet, sit atop a volcano in Hawaii, and send dataover a laser link from the International SpaceStation.

BIO

Sanjay PadhiBIO

Dr. Sanjay Padhi High EnergyPhysics

co-creator of the HTCondor

simulation activities by CMS

before joining CERN

Traci Ruthowski

Traci’s research interests are at the intersectionof applied technology and Earth Sciences.Currently Funded by NSF, Traci is exploring theuse of disruptive technology, including theInternet of Things (IoT), as means to enableresearch in the Alaskan Arctic.Prior to AWS, Traci held several roles at TheUniversity of Michigan including: EnterpriseArchitect for the Office of the Vice President forResearch, HPC Lead for medical research, andSolutions Developer of clinical treatmentsystems. Traci is the author of the book, GoogleVisualizationAPI Essentials.Traci has a Graduate Certification in BusinessAdministration from Eastern Michigan Universityand also holds a BS in Statistics as well as a BFAin Performing Arts Technology from theUniversity of Michigan.

BIO