HPC over CloudHands-on: Cluster automation with Amazon Web Services
Cristian Mezzanotte
17.04.2012
© D
atal
ynx A
G
| S
lide
2| 1
9-A
pr-
12
| C
rist
ian
Mez
zan
ott
e | C
orp
ora
te P
rese
nta
tion
Overview
• Datalynx AG
• Web Services and Cloud
• AWS
• Hands-on!
• Conclusions
© D
atal
ynx A
G
| S
lide
3| 1
9-A
pr-
12
| C
rist
ian
Mez
zan
ott
e | C
orp
ora
te P
rese
nta
tion
Assumptions
• You have a basic knowledge of Cloud Computing concepts
• You know the difference between public, private and hybrid Clouds
• You are familiar with High Performance Computing concepts
© D
atal
ynx A
G
| S
lide
4| 1
9-A
pr-
12
| C
rist
ian
Mez
zan
ott
e | C
orp
ora
te P
rese
nta
tion
Datalynx AG – www.datalynx.com
• Fast growing, customer-focusd, IT company based in Switzerland
• Competences in IT Architectures, Security, Project Management, Quality Management
• Advice. Create. Manage. Secure. Collaborate.
• Customers: Novartis, Roche, Credit Suisse, Swisscom, Adidas, Swiss Post, etc..
© D
atal
ynx A
G
| S
lide
5| 1
9-A
pr-
12
| C
rist
ian
Mez
zan
ott
e | C
orp
ora
te P
rese
nta
tion
Web Services
• What are WSWS are distributed application components in aclient-server model
Do not confuse Web Services with Web Pages Web Pages are for human, Web Services are formachines
• Why use WSInteroperability, available on the WWW, open standards
• How do WS workHTTP, XML, SOAP, WSDL, UDDI
© D
atal
ynx A
G
| S
lide
6| 1
9-A
pr-
12
| C
rist
ian
Mez
zan
ott
e | C
orp
ora
te P
rese
nta
tion
Amazon Web Services
• Public Cloud framework
• Infrastructure as a Service (IaaS)
• Management Console
• API
• Based on Web Services
© D
atal
ynx A
G
| S
lide
7| 1
9-A
pr-
12
| C
rist
ian
Mez
zan
ott
e | C
orp
ora
te P
rese
nta
tion
Amazon Web Services
Elastic Computing Cloud (EC2)
• Create and select new or templeted images of any Operative System, libraries, data, etc..
• Configure network and security settings
• Start, terminate and monitor your instances
• Manage locations, IP endpoints and storage
• Pay-as-you-go
© D
atal
ynx A
G
| S
lide
8| 1
9-A
pr-
12
| C
rist
ian
Mez
zan
ott
e | C
orp
ora
te P
rese
nta
tion
Amazon Web Services
Simple Storage Service (Amazon S3)
• Write, read and delete unlimited objects containing up to 5TB of data
• Data security tools
• Authentication
• Sharing
© D
atal
ynx A
G
| S
lide
9| 1
9-A
pr-
12
| C
rist
ian
Mez
zan
ott
e | C
orp
ora
te P
rese
nta
tion
Amazon Web Services
Cloud Formation
• Template resources
• Create templates of machines and create instances in AWS
• Stack your infrastructure and replicate it into AWS
© D
atal
ynx A
G
| S
lide
10
| 1
9-A
pr-
12
| C
rist
ian
Mez
zan
ott
e | C
orp
ora
te P
rese
nta
tion
Amazon Web Services
EMR
VPC
© D
atal
ynx A
G
| S
lide
11
| 1
9-A
pr-
12
| C
rist
ian
Mez
zan
ott
e | C
orp
ora
te P
rese
nta
tion
Amazon Web Services trends
© D
atal
ynx A
G
| S
lide
12
| 1
9-A
pr-
12
| C
rist
ian
Mez
zan
ott
e | C
orp
ora
te P
rese
nta
tion
The Simulation
• Collaboration with Milan University
• Replica Exchange Molecular Dynamics (REMD) simulation with Amber11
• Pellegrino, Contini et al, Chem. Eur. J 2012, DOI: 10.1002/chem.201104023
© D
atal
ynx A
G
| S
lide
13
| 1
9-A
pr-
12
| C
rist
ian
Mez
zan
ott
e | C
orp
ora
te P
rese
nta
tion
Cluster Instance
Img_remd
/vol
Img_remd
/vol
Img_remd
/vol
Img_remd
/vol
© D
atal
ynx A
G
| S
lide
14
| 1
9-A
pr-
12
| C
rist
ian
Mez
zan
ott
e | C
orp
ora
te P
rese
nta
tion
Big data file transfer service
• The amount of data to be processed and moved is growing fast
• Traditional file transfer tools (ftp, scp..) are inadequate
• Gridftp greatly improves performance up to 300 MB/s (LRZ), 200 MB/s (Teragrid)
© D
atal
ynx A
G
| S
lide
15
| 1
9-A
pr-
12
| C
rist
ian
Mez
zan
ott
e | C
orp
ora
te P
rese
nta
tion
HPC over Cloud vs Traditional HPC
• Availability of resources
• No initial investment on infrastructure
• Pay what you use
• Scalability
• Versatility
• Automation
© D
atal
ynx A
G
| S
lide
16
| 1
9-A
pr-
12
| C
rist
ian
Mez
zan
ott
e | C
orp
ora
te P
rese
nta
tion
Challenges
• Security
• Billing system integration
• Lower performance in some cases
• Licences in some cases
© D
atal
ynx A
G
| S
lide
17
| 1
9-A
pr-
12
| C
rist
ian
Mez
zan
ott
e | C
orp
ora
te P
rese
nta
tionThank you!
Contacts: