amazonweb servicesamazonweb services

AWS 201T WorkshopJames Saull

Solution Architect jsaull@amazon.com

Agenda• House Keeping & Setup

• Introduction to Amazon Web Services

• Lab: Building a Web Property on AWS– Storage: S3, EBS

– Compute

– Content Delivery

– Relational Database

– Dynamo DB

• Lab: Scalability and Availability– Snapshots

– Load Balancing

– Auto Scaling

– Security

• Log Processing Scenario– Logging to AWS

– Elastic Map Reduce

Questions ?

House Keeping• Breaks

Setup

• aws.amazon.com

• Be sure you have permissions and are active

• Firefox or Chrome

• Windows download Putty: www.putty.org

Watch out for unexpected Costs

When the Technical Workshop comes to an end, to avoid unwanted costs:

• Delete your S3 objects

• Destroy your CloudFront distributions

• Stop or Shut Down your EC2 and RDS instances

The customer is responsible for the resources he’s using. AWS declines any responsibility if the customer forgets to shut down resources.

Agenda• House Keeping & Setup

• Introduction to Amazon Web Services

• Lab: Building a Web Property on AWS– Storage: S3, EBS

– Compute

– Content Delivery

– Relational Database

• Lab: Scalability and

Availability– Snapshots

– Load Balancing

– Auto Scaling

– Security

• Log Processing Scenario– Logging to AWS

– Elastic Map Reduce

– Dynamo DB

Who is Amazon.com?Who is Amazon.com?

8

Amazon.com’s Three BusinessesAmazon.com’s Three Businesses[[ ]]

Consumer business

Seller business

IT Infrastructure business

Consumer business

Seller business

IT Infrastructure business

}}

9

Deep experience in

building and operating

global web scale

systems

About Amazon

Web Services

?…get into cloud computing?

How did Amazon…

What is Amazon Web Services?What is Amazon Web Services?

– Reliable, scalable, low-cost infrastructure

– Every service offers APIs

– You can use only the components you need

– All the important “building blocks” are provided

– Reliable, scalable, low-cost infrastructure

– Every service offers APIs

– You can use only the components you need

– All the important “building blocks” are provided

]][

RegionRegion GovCloud RegionGovCloud Region

Global AWS InfrastructureGlobal AWS Infrastructure ]][[

Availability ZoneAvailability Zone

Global AWS InfrastructureGlobal AWS Infrastructure ]][[

Edge LocationsEdge Locations

Global AWS InfrastructureGlobal AWS Infrastructure ]][[

Dallas (2) Dallas (2)

St.LouisSt.Louis

MiamiMiami

JacksonvilleJacksonvilleLos Angeles (2)Los Angeles (2)

Palo AltoPalo Alto

SeattleSeattle

Ashburn

(2)

Ashburn

(2)

NewarkNewarkNew York (3)New York (3)

DublinDublin

London (2)London (2)Amsterdam (2)Amsterdam (2)StockholmStockholm

Frankfurt (2)Frankfurt (2)Paris

(2)

Paris

(2)

Singapore

(2)

Singapore

(2)

Hong Kong

(2)

Hong Kong

(2)

Tokyo (2)Tokyo (2)

Sao PauloSao Paulo

South BendSouth Bend

San JoseSan Jose

OsakaOsakaMilanMilan

Sydney

Hayward

Madrid

The Amazon Web Services universeThe Amazon Web Services universe[[ ]]

amazonweb servicesamazonweb services

The Amazon Web Services universeThe Amazon Web Services universe[[ ]]

SDKSDK

APIAPI

Web ConsoleWeb Console

Command LineCommand Line

[[ ]]

Management +

Interface

Management +

Interface

Cross

Service

Features

Cross

Service

Features

CloudFormationCloudFormationSimple WorkFlowSimple WorkFlow

CloudWatchCloudWatch

Elastic BeanStalkElastic BeanStalk

IAMIAM

The Amazon Web Services universeThe Amazon Web Services universe

(some services are omitted here)(some services are omitted here)

Simple Notification Service

[[ ]]

Management +

Interface

Management +

Interface

Platform

Building

blocks

Platform

Building

blocks

RDSRDS

EMREMR

DynamoDBDynamoDB

CloudFrontCloudFrontSQSSQS

CloudSearchCloudSearchSESSES

Cross

Service

Features

Cross

Service

Features

The Amazon Web Services universeThe Amazon Web Services universe

(some services are omitted here)(some services are omitted here)

RedshiftRedshift

Infrastructure

Building

Blocks

Infrastructure

Building

Blocks

[[ ]]

Management +

Interface

Management +

Interface

Platform

Building

blocks

Platform

Building

blocks

Cross

Service

Features

Cross

Service

Features

GlacierGlacier EBSEBS S3S3Virtual Private CloudVirtual Private Cloud

The Amazon Web Services universeThe Amazon Web Services universe

(some services are omitted here)(some services are omitted here)

EC2

[[ ]]

Management +

Interface

Management +

Interface

RDS CloudSearch

SES CloudFront SQS

EMR DynamoDB

RDS CloudSearch

SES CloudFront SQS

EMR DynamoDB

Elastic Beanstalk

Simple WorkFlow

CloudFormation

CloudWatch

IAM

Elastic Beanstalk

Simple WorkFlow

CloudFormation

CloudWatch

IAM

VPC EC2

EBS S3

VPC EC2

EBS S3

(some services are omitted here)(some services are omitted here)

Infrastructure

Building

Blocks

Infrastructure

Building

Blocks

Platform

Building

blocks

Platform

Building

blocks

Cross

Service

Features

Cross

Service

Features Command Line

Web Console

APIs

SDK

Command Line

Web Console

APIs

SDK

The Amazon Web Services universeThe Amazon Web Services universe

Agenda• House Keeping & Setup

• Introduction to Amazon Web Services

• Lab: Building a Web Property on AWS– Storage: S3, EBS

– Compute

– Content Delivery

– Relational Database

– Dynamo DB

• Lab: Scalability and Availability– Snapshots

– Load Balancing

– Auto Scaling

– Security

• Log Processing Scenario– Logging to AWS

– Elastic Map Reduce

Labs

During this workshop, we will build from scratch a highly available, redundant,

scalable web property on AWS.

Storage

Amazon Simple Storage Service (S3)

Amazon S3 Bucket Volume SnapshotAmazon Elastic

Block Storage

(EBS)

Amazon Elastic Block Storage (Amazon EBS)

Our Building Blocks

Compute

Our Building Blocks

Amazon Elastic Compute Cloud (EC2)

Amazon EC2 Instance Instances AMI DB on

Instance

Instance with

CloudWatch

Elastic IP

Content Delivery & Database

Our Building Blocks

Amazon Database Services RDS/DDB

Amazon RDS MySQL DB Instance DynamoDB

Amazon Cloudfront

Amazon

Cloudfront

Download

Distribution

Edge LocationStreaming

Distribution

console.aws.amazon.com

• Object-based storage

(no Filesystem)

• Easily store/retrieve data

• Durability of 99.999999999% or

99.99%

• Integrated with other AWS

Services

• Scalable

• Redundancy is managed

transparently

• File (Object): up to 5 TB each

• HTTP, HTTPS, BitTorrent

protocols

28

Amazon S3Simple Storage Service

Data

Any Amazon S3 Region

Your Data

Data

Data

DataData

Amazon S3 Redundancy

Data is replicated

multiple times

In case of failure,

data is replicated again,

transparently

• Backup

• Archive (Glacier)

• Disaster Recovery (DR)

• Content Storage

• Website

30

Amazon S3 Use cases

Lab Exercise

• Create an S3 Bucket

• Upload and Download a File

• Check out properties and permissions

Questions ?

• Delivery of content

worldwide

• Static: HTTP / HTTPS

• Streaming: RTMP

• Dynamic Content

• Origin server: S3, EC2,

non-AWS

• Public/Private

Distributions

33

Amazon CloudFrontContent Delivery Network

Regions, Availability Zones, Edge Locations

Dallas

St.Louis Miami

Jacksonville

Los Angeles (2)

Palo Alto

Seattle

Ashburn

Newark

New York (2)

Dublin

London

Amsterdam

Stockholm

FrankfurtParis

Singapore

Hong Kong

Tokyo

Sao Paulo

South Bend

San Jose

38 Edge Locations in total (as of Dec 2012)

Sydney

Let’s simplify a bit:we consider only a few of them

Stockholm

Hong Kong

Sao Paulo

San Jose

Your web servers

in Singapore

Stockholm

Hong Kong

Sao Paulo

San Jose

Content Delivery Network:How it works

Dynamic pages (PHP, Java) (from web servers)

Static content or streaming (with CloudFront)

Your web servers

in Singapore

• Accelerated web content delivery

• Off-load traffic from web servers

• Big spikes in traffic

• Event streaming

• Marketing campaigns

37

Amazon CloudFront Use cases

Lab Exercise

• Create a New Distribution

• Test the Content

Questions ?

• Linux / Windows virtual

machines

• Elastic (scale in minutes)

• Many Instance types

(micro, small, large...)

• AMI: Amazon Machine

Image

• VPC: Virtual Private Cloud

• Security (Groups, Keys...)

• VM Import

• Auto Scaling, Elastic Load

Balancing

40

Amazon EC2Elastic Compute Cloud

Availability ZoneAvailability Zone

I see 20 Availability Zones in total (as of Feb 2013)I see 20 Availability Zones in total (as of Feb 2013)

• Media

• Hosting

• High Performance

Computing

• Dev & Test

• Internal Applications

• Gaming

• ... Everything that

needs computing!

42

EC2 Use cases

Lab Exercise• Create a new Security Group

• Launch an Amazon EC2 instance (Linux)

• Log in with SSH as ec2-user@

• Install a web server

• Create a simple web page

• Test it on a browser: it works!

• Create and attach an Elastic IP

• Create an AMI from an EC2 Instance

Demo Windows• Create a new Security Group for Windows

• Launch a new EC2 instance (Windows)

• Log in with RDP

From Windows: log in with RDP

Questions ?

• Block Level Storage for use

with EC2

• Volume: 1 GB to 1 TB

• Raw unformatted block

device

• Local to an Availability Zone

• Redundant

• Persistent

• Point-in-time snapshots to

Amazon S3

• Integration with CloudWatch

46

EBSElastic Block Storage

Demo EBS• Create a new EBS volume

• Attach it to an EC2 Instance

• Partition it and create a File System

• Mount it; create a file in it

• Create an EBS Snapshot

• Create a new volume from the Snapshot

Questions ?

• Relational Database “as a

Service”

• Simple to Deploy

• Managed by the AWS team

• MySQL, SQLServer or

Oracle (as of 2012)

• Scalable

• Optional: automatic Standby

Replica

• Optional: multiple Read-Only

copies

• Easy DB Snapshots and

automated backup

49

RDSRelational Database Service

Demo RDS• Create a DB Instance on RDS (MySQL)

• Enable Multi-AZ Deployment

• Enable one Read Replica

• Optional: connect to the DB Instance

Questions ?

Agenda• House Keeping & Setup

• Introduction to Amazon Web Services

• Lab: Building a Web Property on AWS– Storage: S3, EBS

– Compute

– Content Delivery

– Relational Database

– Dynamo DB

• Lab: Scalability and Availability– Snapshots

– Load Balancing

– Auto Scaling

– Security

• Log Processing Scenario– Logging to AWS

– Elastic Map Reduce

Let’s take our Web Application to

the next level

EBS

Elastic Load Balancer• Automatically distribute incoming traffic to

multiple Amazon EC2 instances (in the same

Region).

• Automatic Health check

• IPv6 support

• Can be integrated with AutoScaling

Questions ?

Snapshots & AMIs

• Copies of EBS Volumes

• Essential to Reusability

• Copy between Regions

• Durability in S3

Lab Exercise• Duplicate your entire architecture by

making an AMI

• Increase your availability by spreading your application across availability zones

• Bring up an ELB in front of your website

• Optional – Create a CNAME to the ELB

Questions ?

CloudWatch

• Integrated across AWS platform

• Nothing to install

• Custom Metrics

• Set Alarms

• View graphs and statistics

• Feeds into Autoscaling

Lab Exercise• Check out the metrics of your web

application

• Set up an alarm to monitor your site

Questions ?

AutoScaling

• Auto Up and Auto Down

• Runs on CloudWatch metrics

• Notifications via SNS

• Spot or On-demand

• No additional Fees

AutoScaling

• Launch config: AMI to be used

• Autoscaling group: where/how to launch

• Autoscaling policy: what should AS do

• Autoscaling trigger: what will activate AS

Demo Autoscaling

JMETER

Questions ?

Security

• Security Groups

• Granular tiered secure architecture

• Roles for services

• Best Practices - Bastions

Before we get started on the controls…

• AWS Reports, Certifications & Accreditations• SOC 1, Type 2 report

• SOC 2 report

• ISO 27001

• PCI DSS Level 1 service provider

• FISMA Moderate

• MPAA

• Look at http://aws.amazon.com/security

A Protected

Infrastructure:

IAM

VPC

ACL

ELB

AZ

SG

TMG

RDP

WSUS

AV

Lab Exercise• Secure your website by creating a DMZ

between the ELB and your application

• Create a bastion host

Questions ?

Agenda• House Keeping & Setup

• Introduction to Amazon Web Services

• Lab: Building a Web Property on AWS– Storage: S3, EBS

– Compute

– Content Delivery

– Relational Database

– Dynamo DB

• Lab: Scalability and Availability– Snapshots

– Load Balancing

– Auto Scaling

– Security

• Log Processing Scenario– Logging to AWS

– Elastic Map Reduce

Getting your Data into S3

S3Console Upload

FTP

S3 API

AWS Import / Export

Direct Connect

Tsunami UDP

Storage Gateway

3rd Party Commercial

Applications

CloudFrontFlume

AWS Data Pipeline

S3 and Big Data

• Why S3?

• Hadoop Overview

• Hadoop on the Cloud

• Hadoop File System

Questions ?

Introducing Apache Hadoop

• Apache Hadoop• Software for distributed data analysis

• Map/Reduce framework

• Focus on data

• But• Complex

• Hard to setup

• Cap-ex intensive

• Difficult to manage

EMR is Hadoop in the Cloud

Hadoop is an open-source framework for

a cluster of machines

Hadoop is an open-source framework for

parallel processing huge amounts of data on

a cluster of machines

What is Amazon Elastic MapReduce (EMR)?

How does it work?

EMR

EMR ClusterS3

Put the data

into S3

Choose: Hadoop distribution, #

of nodes, types of nodes, custom

configs, Hive/Pig/etc.

Get the output

from S3

Launch the cluster using

the EMR console, CLI, SDK,

or APIs

You can also store

everything in HDFS

011001101

DynamoDB and Big Data

• What is Dynamo?

• Dynamo in Big Data – Volume & Velocity

Questions ?

Watch out for unexpected Costs

When the Technical Workshop comes to an end, to avoid unwanted costs:

• Delete your S3 objects

• Destroy your CloudFront distributions

• Stop or Shut Down your EC2 and RDS instances

The customer is responsible for the resources he’s using. AWS declines any responsibility if the customer forgets to shut down resources.

amazonweb servicesamazonweb services

Thank YouJames Saull

Solution Architect jsaull@amazon.com