aws developer fundamentals

AWS Developer Fundamentals

Josh Padnick October 18, 2014

Desert Code Camp

What We’ll Cover

Forests & Trees

The DetailsThe Big Picture

• What is AWS?

• AWS for Specific Use Cases

• Key Ideas in AWS

The Big Picture

• Running Servers (EC2)

• Storing Data (S3)

The Details

But we only have 60 minutes.So we’ll move fast.

About Me

About Me

Josh Padnickjosh.padnick@gmail.com

http://JoshPadnick.com

• Founded healthcare IT company where we used AWS for 5+ years.• Built three major products for hundreds of thousands of users on AWS. • Hosted 150+ websites on AWS. • Professional AWS Consultant.

602.432.3789

What is AWS?

It’s what Amazon built internally to power their own site.

They architected it so abstractly that it wasn’t even specific to Amazon!

AWS is a suite of IT services used to build or manage software faster, cheaper, and at scale.

• Compute Services give you on-demand virtual machines.

• Storage Services let you store large blocks of unstructured content.

• Database Services allow you to store structured and unstructured data in a variety of ways.

• Networking Services provide technologies for identifying (DNS) resources and connecting resources to on-premises assets.

• Messaging Services enable rich communication scenarios between systems or components.

• Content Delivery Services provide edge locations for frequently accessed content.

• Deployment and Management Services enable packaging, securing, and monitoring AWS applications.

• Big Data Services include tools for ingesting, processing, and storing huge amounts of data.

• Mobile Services include tools for sending push notifications, and streamlining app development.

SOURCE: “AWS Developer Fundamentals” by Richard Seroter. Pluralsight.

Now let’s look at the AWS console.

Primary AWS Competitors

Source: Gartner (July 2014)

Magic Quadrant for Public Cloud Services

Secondary AWS CompetitorsEmail File Sharing DNS / Domain Reg.

So how do you learn about 33 different services?

It kind of reminds me of this place.

• Your first experience is overwhelming.• No one person needs all of Home Depot.• Each person gets just what they need.

AWS is the same.Don’t try to learn everything.Instead, pick your use case, and get to know only the aisles you need.

Use Cases

Major AWS Use Cases

• Scalable web app• Big data• Mobile app support• Backup & storage• Ad hoc (email, DNS, queueing)

AWS Big Data Services

AWS Backup & Storage Services

Today’s Talk is about using AWS with the Scalable Web App.

For the scalable web app, 80%+ of your work will be in just three services.

If you want to explore more about any use case, check out

http://aws.amazon.com/solutions/

I listed just 5 use cases and AWS lists 18! Think of my 5 as the major forests. AWS is just being extra helpful with every tree it can think of.

We’ll go into detail on that shortly. First, let’s talk about:

Key Ideas in AWS

Make your app resilient by using global regions & availability zones

Key Idea #1

INSPIRATION: “AWS Developer Fundamentals” by Richard Seroter. Pluralsight.

AWS Regions Throughout the WorldRegion = Data Center

In the AWS API, use the region’s official name.

Each AWS Region has at least 2Availability Zones.

• Building across regions is very challenging.• Building across availability zones is straightforward.• You can basically purchase (in time and money) the

amount of resilience you want!

Key Idea #1

Pay only for what you need.

Key Idea #2

Almost nothing in AWS has upfront fees.

• There are almost never upfront fees in AWS.• You pay only for what you use.

• EC2 Instances per hour• S3 files per GB stored / transferred• etc.

• You can stop and start instances as you need them before you launch your app.

• You can start with small (or even burstable!) instances, and easily change your instance type later.

Key Idea #2

Architect for failure.

Key Idea #3

http://status.aws.amazon.com/

• Early versions of the AWS docs just stopped short of telling you instances would fail on a periodic basis.

• Instances are now very reliable, but you should still assume they could fail at any time.

• When they inevitably do, this is not “something unexpected”, it’s just another state you expect your infrastructure may enter.

• On the other hand, all AWS services have built in reliability / fault tolerance.

• Note that there have been isolated stories of businesses going under when their AWS account was hijacked. So, it’s always a good idea to keep your most important data on a different AWS account or location.

Key Idea #3

Everything’s an API call away.Sometimes exclusively.

Key Idea #4

• AWS builds their APIs first.• Then they upgrade their AWS Console.• Sometimes the console only implements a subset of the

API!• It would be possible to build a complete AWS console

on your own using only their APIs. Often used for cloud management providers, other partner vendors.

Key Idea #4

Plan to scale out, not up.

Key Idea #5

• AWS does give you many ways to “scale up”.• In the short-term, “scale up” is definitely easier.• But it’s best if you build your architecture to “scale out”• This is most challenging at the database level. Which is

why AWS offers RDS and DynamoDB.• Even if you can’t build perfectly “horizontally scaling”

architectures, you should have that in mind as the ideal.

Key Idea #5

Let’s talk details.

AWS forScalable Web Apps

Let’s describe each of these AWS services.

(We’ll be brief for some)

EC2 Virtual Servers in the Cloud

Use Case:• Launch Linux and Windows servers in the cloud.

EC2

Let’s setup a serverusing non-AWS lingo

Server

ServerHard Drive

ServerHard Drive

Firewall

ServerHard Drive

Firewall

SSH Keys

ServerHard Drive

Firewall

SSH Keys

IP Address

Now let’s use AWS lingo.

EC2 InstanceEBS Volume

Security Group

Key Pairs

Elastic IP

Let’s take these one at a time.

EC2 Instance

They vary along just 3 dimensions

EC2 Instances Come in Multiple Sizes

CPU Memory Network

CATEGORY INSTANCE TYPES

General Purpose

T2, M3

Compute Optimized

C3

Memory Optimized

R3

GPU G2

Storage Optimized I2, HS1

Instance Type

vCPU Memory (GiB)

 Storage (GB)

Networking

Physical Process

Clock Speed

Intel® AES-NI

Intel® AV

Intel® Turbo

EBS OPT

Enhanced t2.micro 1 1 EBS

OnlyLow to

ModeratIntel Xeon

2.5 Yes Yes Yes - -

t2.small 1 2 EBS Only

Low to Moderat

Intel Xeon

2.5 Yes Yes Yes - -

t2.medium

2 4 EBS Only

Low to Moderat

Intel Xeon

2.5 Yes Yes Yes - -

m3.medium

1 3.75 1 x 4 SSD

Moderate

Intel Xeon

2.5 Yes Yes Yes - -

m3.large 2 7.5 1 x 32 SSD

Moderate

Intel Xeon

2.5 Yes Yes Yes - -

m3.xlarge

4 15 2 x 40 SSD

High Intel Xeon

2.5 Yes Yes Yes Yes -

m3.2xlarge

8 30 2 x 80 SSD

High Intel Xeon

2.5 Yes Yes Yes Yes -

c3.large 2 3.75 2 x 16 SSD

Moderate

Intel Xeon

2.8 Yes Yes Yes - Yes

c3.xlarge 4 7.5 2 x 40 SSD

Moderate

Intel Xeon

2.8 Yes Yes Yes Yes Yes

c3.2xlarge

8 15 2 x 80 SSD

High Intel Xeon

2.8 Yes Yes Yes Yes Yes

c3.4xlarge

16 30 2 x 160 SSD

High Intel Xeon

2.8 Yes Yes Yes Yes Yes

c3.8xlarge

32 60 2 x 320 SSD

10 Gigabit

Intel Xeon

2.8 Yes Yes Yes - Yes

g2.2xlarge

8 15 1 x 60 SSD

High Intel Xeon

2.6 Yes - - Yes -

r3.large 2 15.25 1 x 32 SSD

Moderate

Intel Xeon

2.5 Yes Yes Yes - Yes

r3.xlarge 4 30.5 1 x 80 SSD

Moderate

Intel Xeon

2.5 Yes Yes Yes Yes Yes

r3.2xlarge

8 61 1 x 160 SSD

High Intel Xeon

2.5 Yes Yes Yes Yes Yes

r3.4xlarge

16 122 1 x 320 SSD

High Intel Xeon

2.5 Yes Yes Yes Yes Yes

r3.8xlarge

32 244 2 x 320 SSD

10 Gigabit

Intel Xeon

2.5 Yes Yes Yes - Yes

i2.xlarge 4 30.5 1 x 800 SSD

Moderate

Intel Xeon

2.5 Yes Yes Yes Yes Yes

i2.2xlarge

8 61 2 x 800 SSD

High Intel Xeon

2.5 Yes Yes Yes Yes Yes

i2.4xlarge

16 122 4 x 800 SSD

High Intel Xeon

2.5 Yes Yes Yes Yes Yes

i2.8xlarge

32 244 8 x 800 SSD

10 Gigabit

Intel Xeon

2.5 Yes Yes Yes - Yes

hs1.8xlarge

16 117 24 x 2,000

10 Gigabit

Intel Xeon

2 Yes - - - -

CATEGORY INSTANCE TYPES

General Purpose

T2, M3

• When your’e starting out, you can just use the general purpose line.

• The T2 line is especially good for servers that often sit idle, but then need a burst of performance (e.g. low-traffic web servers, build servers, etc.)

Instance Type vCPU Memory (GiB)  Storage (GB)Networking

PerformancePhysical

ProcessorClock Speed

(GHz)

t2.micro 1 1 EBS Only Low to Moderate Intel Xeon family 2.5

t2.small 1 2 EBS Only Low to Moderate Intel Xeon family 2.5

t2.medium 2 4 EBS Only Low to Moderate Intel Xeon family 2.5

m3.medium 1 3.75 1 x 4 SSD ModerateIntel Xeon

E5-2670 v2*2.5

m3.large 2 7.5 1 x 32 SSD ModerateIntel Xeon

E5-2670 v2*2.5

m3.xlarge 4 15 2 x 40 SSD HighIntel Xeon

E5-2670 v2*2.5

m3.2xlarge 8 30 2 x 80 SSD HighIntel Xeon

E5-2670 v2*2.5

EBS Volume

• EBS = Elastic Block Store• You can provision hard drives at the block level,

which means AWS doesn’t care which file system you format it with (e.g. EXT4, ZFX, NTFS)

• You can even create RAID arrays.• If you need extra performance, you can pay for

higher IOPS.

EBS Volumes are basically “virtual hard drives”

You can create EBS volumes directly. But usually, you create

them as part of your EC2 instance.

But you may want to attach multiple EBS volumes to the

same EC2 instance.

Key Pairs

A key pair is just an SSH private key + its corresponding public key.

• You can upload your own keys.• Or AWS creates them for you.• Linux

• Use your key to SSH into the instance• Windows

• Use your key to get the RDP password of the instance

• Best practice is to use a bastion host.• This means you have one instance that is accessible

via SSH from the outside (locked down only to specific IP addresses).

• Once in the bastion host, then you can SSH into other instances.

Security Group

Security Group = Firewall for an Instance

• Create one security group for each “tier” in your app.• You should have a single security group for allowing

“outside access” from specific IPs (the bastion host security group)

• Be paranoid and restrictive. There are lots of bots out there!

Elastic IP

You can get a public IP address from AWS.Then you assign it to any instance!

You can re-assign it later!

Elastic IP

EC2 Instance A EC2 Instance B

Elastic IP

EC2 Instance A EC2 Instance B

X

Elastic IP

EC2 Instance A EC2 Instance B

X

• This means you can re-assign an elastic IP address from a failed instance to a working one.

• Basically, your server and your IP address are no longer bound to each other.

EC2 InstanceEBS Volume

Security Group

Key Pairs

Elastic IP

Now we’re ready to launch an instance!

What’s an AMI?

EBS Volume

Part 2

• We can take snapshots of an EBS volume.• This means we can instantly clone the EBS volume

and attach it to another instance.

EBS Volumes are basically “virtual hard drives”

• We can take snapshots of these EBS volumes, too.• When we take a snapshot of EBS volumes as part

of an EC2 instance, we wind up creating an Amazon Machine Image.

EC2 Instances are “backed” byEBS Volumes

Amazon Machine Image

is used to launch …

EC2 Instance

EBS Volume

• Windows Server 2008 / 2012 • With or without paid Microsoft software

• Multiple Linux distros • Ubuntu• Suse• Amazon Linux

AWS has prepared useful AMIs for us.

Reserved Instances• Use Reserved Instances to save money. These

are a billing concept only; they have no effect on anything else.

• If you can prepay for 1 year, save 40%.

• If you can prepay for 3 years, save 60%

VPC Isolated Cloud Resources

Let’s setup a networkusing non-AWS lingo

Server Server

Subnet B

Server Server

Subnet A

Subnet B

Server Server

Subnet A

“Subnet Firewall” “Subnet Firewall”

Subnet B

Server Server

Subnet A

“Subnet Firewall” “Subnet Firewall”

Network

Now let’s use the AWS lingo

Subnet B

Instance Instance

Subnet A

Network ACL Network ACL

VPC

VPC Recommendations• VPCs are a great way to logically group your instances into

different “clusters”, both for security and management.

• If you can, setup one public (exposed to Internet) subnet each in two different Availability Zones (AZ’s), and one private subnet each two different AZ’s.

• Use Network ACLs for high-level filtering rules (e.g. connecting Subnet A to Subnet B). Instance-level rules have an additional management overhead.

S3Scalable Storage in the Cloud

Use Cases:• Store files in the cloud for low-latency access

S3

Let’s Look at S3.Using AWS lingo.

us-west-2 region us-east-1 region

us-west-2 region us-east-1 region

S3 Bucket “A” S3 Bucket “A”

S3 Bucket “B” S3 Bucket “B”

S3 Buckets• Buckets are “holding tanks” for files and folders.

• Bucket names must be globally unique across an AWS region. For example, you can’t have two buckets named “A” in the us-west-2 region.

• Buckets have properties which govern all files stored in them (examples shortly)

S3 Buckets• Use a bucket as a way to host static websites!

Surprising Things About

S3 Buckets• If enabled, you can preserve, retrieve, and

restore every version of every object stored in this bucket.

• Of course, you also pay to store every version of every object, so tread carefully here.

Cool Things About

S3 Buckets• You can setup “Rules” for a bucket which take effect

on all files or only certain folders in that bucket.

• Example: auto-delete all files X days after they’re created

• Example: automatically move all files to Glacier X days after they’re created.

• Example: first delete, then archive.

Neat Things About

S3 Buckets• You can limit permissions to buckets by IAM

Roles.

• More on IAM in a bit. But for now, note that you can allow only certain instances or certain logged in users to your AWS console to access certain S3 folders.

Helpful Things About

us-west-2 region us-east-1 region

S3 Bucket “A” S3 Bucket “A”

S3 Bucket “B” S3 Bucket “B”

us-west-2 region us-east-1 region

S3 Bucket “A” S3 Bucket “A”

S3 Bucket “B” S3 Bucket “B”

Files and Folders

Files and Folders in S3• Basically works like a standard file system.

• Files can have granular access permission

• Files can have public read permissions or not.

• Files can be accessed with a temporary token so that when a user downloads one in your app, he can’t take that URL and use it again the next day.

Files and Folders in S3• Files can be encrypted server-side by AWS

• You basically check a box indicating you want encryption.

• Then you trust that Amazon actually encrypts it. AWS handles all encryption on their end. You don’t change anything on yours.

• Costs nothing.

• If you want to supply the encryption keys, AWS will support that, too.

Helpful Things About

S3 Recommendations

• Namespace your buckets (e.g. “padnick-dcc14”)

• Intelligently use auto-delete rules to save on cost. If you’re paranoid about needing the files, then archive to Glacier.

• Use very thoughtful folder names in your buckets, then you can apply folder-specific rules. e.g. “builds”, “backup”, “temp”, etc.

• S3 is a great place for key storage (but obviously doesn’t provide key management).

IAMSecure AWS Access Control

• Use IAM to give each member of your team a unique login.

• Never share your root password among everyone!

• You can also use IAM to give permissions to individual EC2 instances for other AWS resources (e.g. S3 buckets)

IAM Recommendations

• Setup your master account, and then put that user/pass in a vault and never give it to anyone!

• Each engineer should have his own IAM login.• Consequences of a bad actor accessing your AWS

account are catastrophic, so please use MFA.• Even if you don’t plan on using IAM roles, create

them and assign them to instances at launch time.

ElastiCache Managed Cache Services

DynamoDB Managed NoSQL Databases

RDS Managed SQL Stores

• This is basically AWS setting up, scaling, and managing your own X for you, where X =

• MySQL, PostrgreSQL, Oracle

• MongoDB

• Redis, Memcached

SESUse AWS as your email service

Summary

• Pick your AWS use case, then dive in.

• EC2, VPC, and S3 are the most popular services.

• Take the time to learn about IAM. It’s not difficult, and will dramatically improve your security posture.

• The best way to learn is by doing!

Josh Padnickjosh.padnick@gmail.com

http://JoshPadnick.com

602.432.3789

Thank you, Now go build something cool!