cloud computing lec2 · 2012. 4. 11. · – salesforce.com • issues – near or impossible to...

Basics of Cloud Computing – Lecture 2

Cloud Providers

Satish Srirama

Outline

• Cloud computing services – recap

• Amazon cloud services

– Elastic Compute Cloud (EC2)

– Storage services - Amazon S3 and EBS– Storage services - Amazon S3 and EBS

• Cloud managers

• Eucalyptus

• SciCloud

11.04.2012 Satish Srirama 2

Cloud Computing – Services - Recap

• Software as a Service – SaaS– A way to access applications

hosted on the web through your web browser

• Platform as a Service – PaaS– A pay-as-you-go model for IT

resources accessed over the

SaaS

Facebook, Flikr, Myspace.com,

Google maps API, Gmail

Level of

Abstraction

resources accessed over the Internet

• Infrastructure as a Service –IaaS– Use of commodity computers,

distributed across Internet, to perform parallel processing, distributed storage, indexing and mining of data

– Virtualization

Satish Srirama

PaaS

Google App Engine,

Force.com, Hadoop, Azure, Amazon S3, etc

IaaSAmazon EC2, SciCloud,

Joyent Accelerators, Nirvanix

Storage Delivery Network, etc.

11.04.2012 3

Cloud Infrastructure

• Provisioning of computing resources: CPU, Memory, Processing

• Basically an “Operating System” on demand

• Usually billed on a per-hour usage model

• Players in this space– Amazon EC2, GoGrid, Joyent, Ubuntu Enterprise Cloud– Amazon EC2, GoGrid, Joyent, Ubuntu Enterprise Cloud

• Management providers: RightScale, ElasticFox, HybridFox

• Operational Differences– Widely different CPU/Memory specs

– Operating System differences

– Difficult/Impossible to move whole images between providers

11.04.2012 4Satish Srirama

Cloud Storage

• Provisioning of data storage: Either file/object based or Database like functionality

• Billed on bandwidth and storage consumed

• Players in the space– Amazon S3, Amazon EBS

– Amazon’s SimpleDB, Google’s BigTable, Apache Cassandra– Amazon’s SimpleDB, Google’s BigTable, Apache Cassandra

• Management Providers: Jungle Disk, Elephant Disk, PutPlace.com

• Issues– Different types of data storage models

– Limitations on the size of individual data units

– Different billing models makes it hard to do a straight comparison

– Access to the data generally uses non-standard query syntax

– No common API


Cloud Platforms

• Provides a complete software stack - An IDE for the cloud

• Takes care of: Runtimes, Load balancing, Resource provisioning

• Players in the space– Google App Engine - Python (initially, now also Java)

– Force.com (SalesForce)

– Microsoft’s Azure - .NET

– Heroku.com (RubyOnRails)– Heroku.com (RubyOnRails)

– Facebook platform

• Issues– Different languages -- Most platforms are unary

– Different operational philosophies• Google App Engine for example doesn’t permit static files. However you can configure it!

– Lots of limitations in terms of deployment

– Completely reliant on the provider for complete uptime and operation

– Widely different billing models


Cloud Applications

• Applications that are completely ‘online’

• Operate on data that is stored in the ‘cloud’ or ‘ether’

• No client software generally required

• Billing: Ad. Revenue, Premium Services

• Players in this space– Google Apps - Gmail / Google Docs– Google Apps - Gmail / Google Docs

– Apple’s MobileMe

– Microsoft’s Live - Hotmail, Live Spaces

– SalesForce.com

• Issues– Near or impossible to move between providers

• GMail to Hotmail requires major disruption

– End user focused• Consumer side of cloud computing

– Completely reliant on the provider for complete uptime and operation


Other cloud services

• Provides services, which other applications can utilize

• Specific to vertical markets where most “Web2.0” standards live

• Usually free for non-commercial use

• Players in the space– Google/Yahoo Maps

– PayPal / Google Checkout– PayPal / Google Checkout

– Google / Yahoo WebServices

– Amazon Merchant Services

– Amazon Simple Queue Service

• Issues/Comments– Some “Web2.0” services have attempted a standardization path

– Most however are complete vendor lock-in

– Mashup applications utilize these Cloud Service


Providers we focus at

• Amazon Web Services

– Amazon EC2

– Amazon S3

– Amazon EBS

• Eucalyptus• Eucalyptus

• SciCloud

• Management providers

– ElasticFox

– RightScale


Amazon Elastic Compute Cloud (EC2)

• One of the very early pioneers of cloud computing

• In a nutshell:

– On Demand “Operating System”

• Complete virtual computer with CPU, Memory and disk spacespace

– Based on the XEN virtual image platform

• Variety of operating systems available

– Linux (Fedora, Ubuntu, CentOS, etc)

– Open Solaris

– Microsoft Windows


EC2 continued…

• Very simple pricing structure– CPU hours

– Machine size

– Bandwidth in and out of cloud

• Extremely FAST start up• Extremely FAST start up– 2-3 minutes from start to finish

• Instance snapshotting

• Very large/generous disk space provisioning– 160GB minimum for the standard instances

• Flexible API to control everything

• Wide range of virtual machine types


EC2 Instance types*

Instance CPU Memory Storag

e

Platform API Name Price (per h.)

Small

(default)

1 EC2

computing

unit

1.7 GB 160 GB 32 bit m1.small $0.08 (*nix)

$0.115 (win)

Medium 2 EC2 units 3.75 GB 410 GB 32 bit m1.medium $0.16 (*nix)

$0.23 (win)$0.23 (win)

Large 4 EC2 units 7.5 GB 850 GB 64 bit m1.large $0.32 (*nix)

$0.46 (win)

Extra Large 8 EC2 units 15 GB 1 690

GB

64 bit m1.xlarge $0.64 (*nix)

$0.92 (win)

Micro Up to 2 EC2

units

613 MB EBS

storage

only

32/64 bit t1.micro $0.02 (*nix)

$0.03 (win)

*Data taken on 09.04.2012


http://aws.amazon.com/ec2/instance-types/

EC2 advanced Instance types*

Instance CPU Memor

y

Storag

e

Platfor

m

API Name Price (per h.)

High-Memory

On-Demand

Instances

6.5 - 26

EC2

units

17.1 –

68.4 GB

420 -

1690

GB

64 bit m2.xlarge

m2.2xlarge

m2.4xlarge

Available at:

http://aws.a

mazon.com/

ec2/pricing/

High-CPU 5, 20 1.7, 7 350, 64 bit c1.medium Same as High-CPU

Instances

5, 20

EC2

units

1.7, 7

GB

350,

1690

GB

64 bit c1.medium

c1.xlarge

Same as

above

Cluster

Compute

Instances

33.5, 82

EC2

units

23, 60.5

GB

1690,

3370

GB

64 bit cc1.4xlarge

cc2.8xlarge

Same as

above

Cluster GPU

Instances

33.5

EC2

units

22 GB 1690

GB

64 bit cg1.4xlarge Same as

above

*Data taken on 09.04.2012


http://aws.amazon.com/ec2/instance-types/

EC2 Terminology

• Having an account– Access Key

– Secret Key

– Security group

– Availability Zone

• Amazon Machine Image (AMI)– A Virtual Machine File– A Virtual Machine File

– ami-XXXX

– Stored in a special bucket in Amazon’s S3

• Public and Private instances available– Private instances incur only S3 storage costs

• A Running Machine– Amazon Instance ( i-XXXX )

– Booted/Created from an Amazon Image

• Elastic IP addresses


Troubles with EC2

• On power-off all hard disk data is lost

• IP addresses are assigned at random

• Can’t turn off public IP address

• Do not forget to terminate the instances• Do not forget to terminate the instances


Simple Storage Service (S3)

• Enables you to upload, download, and store data across the Internet

• Buckets store data

– Buckets are the fundamental container in Amazon S3 for data storagedata storage

– 100 buckets for account

– No limit on no of objects that can be stored in a bucket

– Can store up to 5 TB of data in one object

– Object stores Data and Metadata

– Objects stored in a Region never leave the Region

– You cannot modify or append data to an existing object


Simple Storage Service (S3) - continued

• Objects are retrieved via a unique, developer-assigned key

• Keys– Example: http://doc.s3.amazonaws.com/2006-03-01/AmazonS3.wsdl

"doc" is the bucket name and "2006-03-01/AmazonS3.wsdl" is the key.

• Prices– Storage: $0.125 per GB/Month–

– Reduced Redundancy Storage : $0.093 per GB/Month

– Data Transfer in: $0.12/GB to transfer data

– PUT, COPY, POST, or LIST Requests: $0.01 per 1,000 Requests

– GET requests: $0.01 per 10,000 Requests

• Download Data– Downloading is possible from any where

– You can enable others to download and can charge them (Using Amazon DevPay)


http://aws.amazon.com/s3/

Elastic Block Storage (EBS)

• The answer to the persistence problem

• Raw unformatted external block devices

• Allocate 1GB to 1TB volumes

• 20 volumes per account

• Format with your own choice of file system • Format with your own choice of file system

• Attach to running instances in the same availability zone

• Automatically replicated to prevent data loss

• Create snapshots for backup, or to create new volumes from


Basic ec2-api-tools

• ec2-describe-images

• ec2-describe-instances

• ec2-run-instances

• ec2-create-volume• ec2-create-volume

• ec2-attach-volume

• ec2-allocate-address

• ec2-associate-address

• ec2-terminate-instances


Other cloud services from Amazon

• AWS management console

• Amazon Simple Queue Service (SQS)

• Amazon DynamoDB

• Amazon Relational Database Service• Amazon Relational Database Service

• Amazon CloudWatch

• Amazon Elastic MapReduce

• Elastic Load Balancing

• etc.


Getting started

• Signup for an AWS account

– Use your existing Amazon.com account if you

want

• Register a credit card• Register a credit card

– Billed on the 1st of every month for previous

month usage

• http://aws.amazon.com/ec2


Getting started – for free

• http://aws.amazon.com/free/ - free tier

• Valid for 12 months

• 750 hours of EC2 Linux Micro Instance usage

• 5 GB of S3 standard storage, 20,000 Get Requests, and 2,000 Put Requests2,000 Put Requests

• 30 GB of Amazon EBS

• 25 Amazon SimpleDB Machine Hours and 1 GB of Storage

• Be careful

– You cross the limits, you are charged without any notice

11.04.2012 Satish Srirama 23/35

ElasticFox

• Mozilla Firefox extension for managing your Amazon EC2 account

• Supported services

– List available AMIs – List available AMIs

– List your running instances

– Launch new instances of an AMI

– Manage security groups and launch permissions associated with your instances

– Manage Amazon EBS volumes


ElasticFox - GUI


Independent software vendors (ISV)

for S3

S3 Explorer filicio.us Jungle Disk


S3 Firefox Organizer MyOwnDB

RightScale

• RightScale cloud management platforms

• Automated configuration– From boot to production on auto-pilot

– Server templates & Right scripts

– Cloning/ replication of set-ups– Cloning/ replication of set-ups

• Automated management– Monitoring, alerts, SLAs -> actions

• Have several examples

• Can login as a developer https://my.rightscale.com


Eucalyptus

• Are you OK with using your credit card?

• Open source project

• Elastic Utility Computing Architecture Linking Your Programs To Useful Systems

• Web services based implementation of EC2 • Web services based implementation of EC2 infrastructure

• For establishing private clouds

• Functions as a software overlay– Existing installation should not be violated (too much)

• Focus on installation and maintenance


Eucalyptus Usage

• Foster greater understanding and uptake of cloud computing – Provide a vehicle for extending what is known about the utility model

of computing

• Experimentation vehicle prior to buying commercial services– Provide development, debugging, and “tech preview” platform for

Public Clouds

• Homogenize local IT environment with Public Clouds• Homogenize local IT environment with Public Clouds– AWS functionality locally makes moving using Amazon AWS easier,

cheaper, and more sustainable

• Provide a basic software development platform for the open source community– E.g. the “Linux Experience”

• Not designed as a replacement technology for AWS or any other Public Cloud service


Eucalyptus - architecture

Amazon EC2 Interface


Basic euca2ools

• euca-describe-images

• euca-describe-instances

• euca-run-instances

• euca-create-volume• euca-create-volume

• euca-attach-volume

• euca-terminate-instances

• euca-describe-availability-zones


Other private cloud technologies

• Globus/Nimbus– Client-side cloud-computing interface to Globus-enabled TeraPort

cluster at University of Chicago

– Based on GT4 and the Globus Virtual Workspace Service

– Shares upsides and downsides of Globus-based grid technologies

• Enomalism (now called ECP (Enomaly's Elastic Computing Platform))– Start-up company distributing open source – Start-up company distributing open source

– REST APIs

• Reservoir– European open cloud project

– Many layers of cloud services and tools

– Ambitious and wide-reaching but not yet accessible as an implementation


Scientific Computing Cloud (SciCloud)

• Distributed Systems Group owned private cloud infrastructure

• Eucalyptus setup

• Goal of the project• Goal of the project

– To Establishing a private cloud at universities

– To efficiently use the already existing resources of universities

– To address computationally intensive scientific, mathematical, and academic problems


http://ds.cs.ut.ee/research/scicloud

SciCloud architecture

HPC Cluster

SciCloud controller

Kuu Cluster

Amazon EC2 Interface

Web browser SOAP/REST based tools

Cluster controller


katel40

SciCloud controller

Cluster controller Storage controllerkuu1

Node controller

Node controller

Katel1-39

Custom

images

Cluster controller

Node controllerkuu2

Node controllerkuu8

katel42

2

0

SciCloud - continued

• For further information (SciCloud homepage)

– http://ds.cs.ut.ee/research/scicloud

• Entry link

– https://scicloud1.mt.ut.ee:8443– https://scicloud1.mt.ut.ee:8443

– Follow the manuals for detailed information

– Description of the images

– How to run instances

– How to create customized images

– Work with them in the lab exercises


SciCloud – Research topics

• Have customized images supporting

– Scientific Computing• NumPy

• SciPy

• Enterprise computing• Enterprise computing

– Ongoing research with auto scaling

– Ongoing research with distributed troubleshooting

– Porting enterprise application onto the cloud

– Ongoing research with load balancing

• MapReduce (Lecture 3)

– Reducing scientific computing problems to MapReduce


SciCloud – Research topics - continued

• Mobile Cloud

– Accessing cloud services from the mobiles

– Building social network and mashup applications on Android/iOS devices

• Several theses are open…• Several theses are open…

– Check at project home page or http://ds.cs.ut.ee/theses

– Interested students can approach me

– Open to new research ideas

• More discussion in Lecture 7


Any Questions?


Further topics to discuss

• Will try to discuss other cloud providers in 6th

lecture

• I might focus at

– Aneka framework– Aneka framework

– Google AppEngine

– SalesForce

• Next lecture

– MapReduce

– Hadoop


References

• Amazon Web (Cloud) Services –documentation http://aws.amazon.com/documentation/

• Trying the basic example with RightScalehttp://support.rightscale.com/03-Tutorials/02-AWS/01-Beginner_Examples/3._Basic_ExampleBeginner_Examples/3._Basic_Example

• SciCloud – homepage http://ds.cs.ut.ee/research/scicloud

• Eucalyptus User's Guide http://open.eucalyptus.com/wiki/EucalyptusUserGuide

• TODO:

– Work with SciCloud instances


cloud computing lec2 · 2012. 4. 11. · – salesforce.com • issues – near or impossible to...

Documents