s oftware t ools for a cademics and r esearchers massachusetts institute of technology office of...
Post on 20-Dec-2015
214 views
TRANSCRIPT
![Page 1: S oftware T ools for A cademics and R esearchers MASSACHUSETTS INSTITUTE OF TECHNOLOGY Office of Educational Innovation and Technology](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a232f4/html5/thumbnails/1.jpg)
Software Tools for Academics and Researchers http://web.mit.edu/starMASSACHUSETTS INSTITUTE OF TECHNOLOGY
Office of Educational Innovation and Technology
StarCluster
Justin Riley ([email protected])
Software Tools for Academics and ResearchersOffice of Educational Innovation and Technology
Massachusetts Institute of Technology77 Massachusetts Ave.Cambridge, MA 02139
HPC on Amazon's Elastic Compute Cloud
(http://web.mit.edu/starcluster)
![Page 2: S oftware T ools for A cademics and R esearchers MASSACHUSETTS INSTITUTE OF TECHNOLOGY Office of Educational Innovation and Technology](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a232f4/html5/thumbnails/2.jpg)
Software Tools for Academics and Researchers http://web.mit.edu/starMASSACHUSETTS INSTITUTE OF TECHNOLOGY
Office of Educational Innovation and Technology
Outline About STAR Overview of Amazon Web Services (AWS) Elastic Compute Cloud (EC2) Hardware Motivations Behind StarCluster About StarCluster StarCluster Features StarCluster Advantages StarCluster Live Demo EC2 Performance Materials Science Research Case Study
![Page 3: S oftware T ools for A cademics and R esearchers MASSACHUSETTS INSTITUTE OF TECHNOLOGY Office of Educational Innovation and Technology](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a232f4/html5/thumbnails/3.jpg)
Software Tools for Academics and Researchers http://web.mit.edu/starMASSACHUSETTS INSTITUTE OF TECHNOLOGY
Office of Educational Innovation and Technology
About STARWhat's your biggest problem bringing your research into the classroom?
![Page 4: S oftware T ools for A cademics and R esearchers MASSACHUSETTS INSTITUTE OF TECHNOLOGY Office of Educational Innovation and Technology](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a232f4/html5/thumbnails/4.jpg)
Software Tools for Academics and Researchers http://web.mit.edu/starMASSACHUSETTS INSTITUTE OF TECHNOLOGY
Office of Educational Innovation and Technology
Elastic Compute Cloud (EC2) Features:
Amazon EC2 allows you to dynamically allocate and terminate Linux “virtual” machines with a variety of hardware configurations
Pay only for what you use (i.e. machine hours and data transfer)
Ability to capture software configurations into Amazon Machine Images (AMI) for later use.
AMI's can be used to launch multiple machines with identical software configurations.
Overview of
![Page 5: S oftware T ools for A cademics and R esearchers MASSACHUSETTS INSTITUTE OF TECHNOLOGY Office of Educational Innovation and Technology](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a232f4/html5/thumbnails/5.jpg)
Software Tools for Academics and Researchers http://web.mit.edu/starMASSACHUSETTS INSTITUTE OF TECHNOLOGY
Office of Educational Innovation and Technology
Elastic Block Storage (EBS) Features:
EBS volumes are highly available, highly reliable volumes that can be attached to a running Amazon EC2 machine and are exposed as standard block devices
Allows you to create point-in-time snapshots of your data.
Pay per month based on allocation as well as per 1 million I/O requests ($0.10/GB allocated/month and $0.10/million I/O requests)
1GB-1TB limit per EBS volume
Overview of
![Page 6: S oftware T ools for A cademics and R esearchers MASSACHUSETTS INSTITUTE OF TECHNOLOGY Office of Educational Innovation and Technology](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a232f4/html5/thumbnails/6.jpg)
Software Tools for Academics and Researchers http://web.mit.edu/starMASSACHUSETTS INSTITUTE OF TECHNOLOGY
Office of Educational Innovation and Technology
Elatic Compute Cloud Hardware
Instance Arch CPU RAM Storage I/O Performance Cost/hr
Small 32bit 1.0-1.2GHz 1.7GB 160GB Moderate $0.10/hr
Large 64bit 2.0-2.4GHzdual-core
7.5GB 860GB High $0.40/hr
Extra Large 64bit 2.0-2.4GHz quad-core
15GB 1.690TB High $0.80/hr
Instance Arch CPU RAM Storage I/O Performance Cost/hr
Medium 32bit 2.5-3.0GHzdual-core
1.7GB 350GB Moderate $0.20/hr
Extra Large 64bit 2.5-3.0GHzquad-core(ht)
7GB 1.690TB High $0.80/hr
Standard Instances
High CPU Instances
One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.
![Page 7: S oftware T ools for A cademics and R esearchers MASSACHUSETTS INSTITUTE OF TECHNOLOGY Office of Educational Innovation and Technology](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a232f4/html5/thumbnails/7.jpg)
Software Tools for Academics and Researchers http://web.mit.edu/starMASSACHUSETTS INSTITUTE OF TECHNOLOGY
Office of Educational Innovation and Technology
Motivations Behind StarCluster
StarHPC - an on demand compute cluster for parallel programming with both OpenMP and OpenMPI technologies. It provides a virtual desktop environment, hosted on EC2, configured with all the necessary tools for programming in OpenMP/OpenMPI.
http://web.mit.edu/star/hpc
StarMolsim - a web application used to run materials modeling research software. It enables the user to run various simulations on a distributed compute cluster and retrieve the results, all from a web browser.
http://web.mit.edu/star/molsim
![Page 8: S oftware T ools for A cademics and R esearchers MASSACHUSETTS INSTITUTE OF TECHNOLOGY Office of Educational Innovation and Technology](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a232f4/html5/thumbnails/8.jpg)
Software Tools for Academics and Researchers http://web.mit.edu/starMASSACHUSETTS INSTITUTE OF TECHNOLOGY
Office of Educational Innovation and Technology
HPC in the Classroom
Constantinos Evangelinos,PhD
Researcher in Earth, Atmospheric, and Planetary Sciences.
Nicola Marzari, PhD
Associate Professor in Computational Materials Science
Markus Buehler, PhD
Assistant Professor in Civil and Environmental Engineering
Materials Modeling Parallel Programming
![Page 9: S oftware T ools for A cademics and R esearchers MASSACHUSETTS INSTITUTE OF TECHNOLOGY Office of Educational Innovation and Technology](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a232f4/html5/thumbnails/9.jpg)
Software Tools for Academics and Researchers http://web.mit.edu/starMASSACHUSETTS INSTITUTE OF TECHNOLOGY
Office of Educational Innovation and Technology
StarHPCUse case: students have direct access to a HPC cluster to actively develop parallel programs using the Message Passing Interface (MPI)
StarHPC was used for 2 weeks in an Independent Activities Period (IAP) course for parallel programming using OpenMP and OpenMPI.
Result: Creating a 4-node cluster for two weeks came out to about $25 per student using Amazon EC2.
SSH/VNC
Virtual Desktop EnvironmentEC2 OpenMPI ClusterUser
![Page 10: S oftware T ools for A cademics and R esearchers MASSACHUSETTS INSTITUTE OF TECHNOLOGY Office of Educational Innovation and Technology](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a232f4/html5/thumbnails/10.jpg)
Software Tools for Academics and Researchers http://web.mit.edu/starMASSACHUSETTS INSTITUTE OF TECHNOLOGY
Office of Educational Innovation and Technology
StarMolsimUse case: students log in to a web application as a proxy to the computing resources. The web application handles communicating with the cluster to submit jobs, retrieve the results, etc.
Result: Amazon EC2 was used to replace a traditional 9-node HPC cluster for an entire semester. The cost for using the 9 node EC2 cluster for the semester was around $3,000-$4,000.
UserEC2 Cluster
Web Server hosting GenePattern from the Broad Institute of MIT and Harvard
![Page 11: S oftware T ools for A cademics and R esearchers MASSACHUSETTS INSTITUTE OF TECHNOLOGY Office of Educational Innovation and Technology](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a232f4/html5/thumbnails/11.jpg)
Software Tools for Academics and Researchers http://web.mit.edu/starMASSACHUSETTS INSTITUTE OF TECHNOLOGY
Office of Educational Innovation and Technology
About StarCluster
http://web.mit.edu/stardev/cluster
StarCluster is a utility for creating and managing general purpose compute clusters hosted on Amazon's Elastic Compute Cloud (EC2).
StarCluster makes it easy for a user to create their own compute cluster on EC2 and pay only for what they use.
Software included in the virtual machine: OpenMPI NFS'd /home directory Sun Grid Engine Scipy/Numpy/IPython Compilers for installing your own custom software Ubuntu Linux OS with apt-get for installing additional OS software
StarCluster Dependencies: Registered and fully configured EC2 account. Python 2.4+ Paramiko library for Python
![Page 12: S oftware T ools for A cademics and R esearchers MASSACHUSETTS INSTITUTE OF TECHNOLOGY Office of Educational Innovation and Technology](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a232f4/html5/thumbnails/12.jpg)
Software Tools for Academics and Researchers http://web.mit.edu/starMASSACHUSETTS INSTITUTE OF TECHNOLOGY
Office of Educational Innovation and Technology
StarCluster Features
• Simple configuration with sensible defaults
• One command to create and configure a n-node cluster on EC2
• Utilizes Amazon's Elastic Block Storage to store and
snapshot your applications and data. • Easily recreate identical working environments
• 32bit/64bit Ubuntu 9.04 public AMI's
![Page 13: S oftware T ools for A cademics and R esearchers MASSACHUSETTS INSTITUTE OF TECHNOLOGY Office of Educational Innovation and Technology](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a232f4/html5/thumbnails/13.jpg)
Software Tools for Academics and Researchers http://web.mit.edu/starMASSACHUSETTS INSTITUTE OF TECHNOLOGY
Office of Educational Innovation and Technology
StarCluster Features
• Automatic Configuration of:
o Sun Grid Engine with Parallel Environment (PE)
o OpenMPI with SGE PE Support
o NFS shares (e.g. /home and /opt)
o Passwordless SSH
o ~147GB local scratch space on /scratch for each node
![Page 14: S oftware T ools for A cademics and R esearchers MASSACHUSETTS INSTITUTE OF TECHNOLOGY Office of Educational Innovation and Technology](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a232f4/html5/thumbnails/14.jpg)
Software Tools for Academics and Researchers http://web.mit.edu/starMASSACHUSETTS INSTITUTE OF TECHNOLOGY
Office of Educational Innovation and Technology
StarCluster Advantages• Portable, launch a cluster from virtually anywhere!
• Supplements existing resources when needed
• Easily store your applications and data in the cloud via EBS. Simply upload your applications/data to /home and your data will be available each time you launch StarCluster.
• Easy to install additional OS software. Just launch the AMI, use the package manager to install additional software, and rebundle the AMI to create your own customized version of StarCluster
• Easily package results of computational experiment for reproducible research
![Page 15: S oftware T ools for A cademics and R esearchers MASSACHUSETTS INSTITUTE OF TECHNOLOGY Office of Educational Innovation and Technology](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a232f4/html5/thumbnails/15.jpg)
Software Tools for Academics and Researchers http://web.mit.edu/starMASSACHUSETTS INSTITUTE OF TECHNOLOGY
Office of Educational Innovation and Technology
StarCluster Live Demo
![Page 16: S oftware T ools for A cademics and R esearchers MASSACHUSETTS INSTITUTE OF TECHNOLOGY Office of Educational Innovation and Technology](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a232f4/html5/thumbnails/16.jpg)
Software Tools for Academics and Researchers http://web.mit.edu/starMASSACHUSETTS INSTITUTE OF TECHNOLOGY
Office of Educational Innovation and Technology
EC2 PerformanceSummary:
Message Passing – extremely poor in comparison to local HPC resources.
Embarrassingly Parallel – much better, but still under performs compared to local HPC resources.
Walker, E. (2008, October) benchmarking Amazon EC2 for high-performance scientific computing.
Retrieved from http://www.usenix.org/publications/login/2008-10/openpdfs/walker.pdf
![Page 17: S oftware T ools for A cademics and R esearchers MASSACHUSETTS INSTITUTE OF TECHNOLOGY Office of Educational Innovation and Technology](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a232f4/html5/thumbnails/17.jpg)
Software Tools for Academics and Researchers http://web.mit.edu/starMASSACHUSETTS INSTITUTE OF TECHNOLOGY
Office of Educational Innovation and Technology
Materials Science Case Study54 relaxation calculations, 25 and 32 atoms (C,N,O,H), standard convergence criterion (Espresso-4.0.5 with MKL 10 and gfortran/gcc 4.3.3)
Worst : 53%Best : 57%
Worst : 58%Best : 69%
Worst Best
0
1
2
3
4
5
6
7
8
9
CPU Time on a 4 node (c1.xlarge) EC2 Cluster
OT1
EC2
CP
U T
ime
(h
rs)
Worst Best
0
2
4
6
8
10
12
Wall Time on a 4 node (c1.xlarge) EC2 Cluster
OT1
EC2
Wa
ll T
ime
(h
rs)
![Page 18: S oftware T ools for A cademics and R esearchers MASSACHUSETTS INSTITUTE OF TECHNOLOGY Office of Educational Innovation and Technology](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a232f4/html5/thumbnails/18.jpg)
Software Tools for Academics and Researchers http://web.mit.edu/starMASSACHUSETTS INSTITUTE OF TECHNOLOGY
Office of Educational Innovation and Technology
Getting Started with StarCluster
Point your web browser to http://web.mit.edu/starcluster
![Page 19: S oftware T ools for A cademics and R esearchers MASSACHUSETTS INSTITUTE OF TECHNOLOGY Office of Educational Innovation and Technology](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a232f4/html5/thumbnails/19.jpg)
Software Tools for Academics and Researchers http://web.mit.edu/starMASSACHUSETTS INSTITUTE OF TECHNOLOGY
Office of Educational Innovation and Technology
AWS Funding Opportunities...
Teaching Grants for educators using AWS in courses (plus access to selected course content resources)
Research Grants for academic researchers using AWS in their work
Project Grants for student organizations pursuing entrepreneurial endeavors; Tutorials for students that want to use AWS for self-directed learning
Solutions for university administrators looking to use cloud computing to be more efficient and cost-effective in the university's IT Infrastructure
http://aws.amazon.com/education/
The AWS in Education program offers:
Learn more about AWS in Education programs:
![Page 20: S oftware T ools for A cademics and R esearchers MASSACHUSETTS INSTITUTE OF TECHNOLOGY Office of Educational Innovation and Technology](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a232f4/html5/thumbnails/20.jpg)
Software Tools for Academics and Researchers http://web.mit.edu/starMASSACHUSETTS INSTITUTE OF TECHNOLOGY
Office of Educational Innovation and Technology
Acknowledgements
Professor Buehler (MIT)
Professor Marzari (MIT)
Constantinos Evangelinos (MIT)
Nicolas Poilvert
Nicolas Pinto (MIT)
Amazon Web Services
![Page 21: S oftware T ools for A cademics and R esearchers MASSACHUSETTS INSTITUTE OF TECHNOLOGY Office of Educational Innovation and Technology](https://reader030.vdocuments.us/reader030/viewer/2022032800/56649d485503460f94a232f4/html5/thumbnails/21.jpg)
Software Tools for Academics and Researchers http://web.mit.edu/starMASSACHUSETTS INSTITUTE OF TECHNOLOGY
Office of Educational Innovation and Technology
Thanks for coming!
Any questions?