effective ec2

Effective EC2PyCon Italia Qu4ttro - May 7/8/9 2010

Valentino Volonghi - [email protected]

Wednesday, May 19, 2010

mailto:[email protected]

mailto:[email protected]

Buzz

Effective EC2 - Valentino Volonghi

Why I’ll never own another server...Joe Stump - Digg Lead Architect, CTO SimpleGeo

http://stu.mp/2010/04/why-ill-never-own-another-server.html

Cloud computing economies of scaleJames Hamilton - VP & Distinguished Engineer, Amazon Web Services

http://live.visitmix.com/MIX10/Sessions/EX01

Moving to the cloudReddit Team

http://blog.reddit.com/2009/11/moving-to-cloud.html








Buzzwords

Agility

Cost

Location Independence

Multi-tenancy

Reliability

Scalability

Security Maintenance Metering

Source: WikipediaEffective EC2 - Valentino Volonghi


Competition...



Google AppEngine

• Transparently scalable

• No maintenance headaches

• Software must be written ad hoc

• Little control over infrastructure



Rackspace Cloud

• CloudServers/CloudFiles/CloudSites

• Some nice additional features

• Fewer datacenters and can’t pick them

• 12/11/07, 09/07/08, 03/11/09



Others



Amazon AWS

• 4 independent regions - 10 data centers

• 3rd party vendor support

• Well integrated services accessible via API

• “Poor” single-node performance

• Flexible upper limits



Amazon AWS

Source: www.businessweek.com/technology/content/apr2010/tc20100428_085106.htmEffective EC2 - Valentino Volonghi

Amazon Web Services can be as bigas our retail business

Andy Jassy - Amazon Senior VP Cloud Computing Business

Revenue: $650.000.000

Market Share:77%


http://www.businessweek.com/technology/content/apr2010/tc20100428_085106.htm

http://www.businessweek.com/technology/content/apr2010/tc20100428_085106.htm

AdRoll

• Scaling High ROI Display Advertising

• 60M paid campaigns impressions

• 200M advertiser pixels impressions

• 99% of requests below 3ms think time

• Realtime Dynamic Ads



AdRoll 2008

• ServePath Housing

• NaviSite CDN with mod_python

• Self-Hosted DNS

• MySQL DB

• NFS



AdRoll 2009

• ServePath Housing

• Amazon EC2 auto-managed AdServers

• Amazon S3/CloudFront CDN

• Dynect DNS Global Server Load Blancing

• MySQL DB

• NFS



AdRoll 2010

• Fully Amazon EC2

• Akamai with Amazon S3 as origin

• Dynect DNS GSLB

• PostgreSQL DB

• Amazon S3 instead of NFS



Akam

ai

S3

AdR

oll.c

om

Log

Proc

essi

ng

DB

Sche

dulin

gSy

ncin

g

3rd party platform

sA

dServers

Plan

Logs/Performance

Campaigns/Reports/Plan

Ads

Plan/Config



http://www.adroll.com

http://www.adroll.com

AdRoll 2010


Network Uptime: 100%

Avg Resp Time: 96ms

Measured by Pingdom


MigrationPhase I - May 2009



New Ad Servers

• Very Quick & faultless boot procedure

• Easy software upgrade

• Low latency network

• Real time monitoring



New Ad Servers

• Custom bundled AMI stored in S3

• Python deploys the AdServer on boot

• Hidden real load test

• Boto monitoring



Boto monitoringdef regional_max_allocation(start_adserver=ec2.Image.start_adserver): zones = [zone.name for zone in ec2.bconn.get_all_zones()]

(availables_with_ip, availables_without_ip, software_failures, startings) = get_instances_by_status()

def setup_candidate(failure=None, free_ip=None): if availables_without_ip: new_instance = get_instance_in_unused_zone(availables_with_ip, availables_without_ip, lenient=True) if failure is not None: free_ip = failure.disassociate_elastic()

availables_without_ip.remove(new_instance) availables_with_ip.add(new_instance) new_instance.associate_elastic(free_ip) return

if startings: startings.pop() return

start_adserver("adserver-" + utils.uuid(), zone=get_unused_zone(availables_with_ip, zones))

for software_failure in software_failures: setup_candidate(failure=software_failure)

free_ips = ec2.get_all_free_elastic_ip(default=[]) for free_ip in free_ips: setup_candidate(free_ip=free_ip)Effective EC2 - Valentino Volonghi


MigrationPhase 2 - March 2010



Preparation

• SQLAlchemy is awesome

• PostgreSQL 5-10x faster than MySQL

• 1 EBS is slower than 8 in RAID!! :)

• m1.small instances are useless



Automated Deploy

• Stock Ubuntu AMI

• Setup scripts on S3

• Fabric

➡ 5 minutes to boot a new instance

➡ No maintenance overhead



Automated Deploy#!/bin/bash -ex

wget -O- run.alestic.com/install/runurl | bashrunurl "https://$base_s3/setup/add-keys-and-cert?SIGNATURE"runurl $base_s3/machine/dbrunurl $base_s3/deploy/db

Deploy DB

#!/bin/bash -ex

wget -O- run.alestic.com/install/runurl | bashrunurl "https://${base_s3}/setup/add-keys-and-cert?SIGNATURE"runurl $base_s3/machine/webrunurl $base_s3/deploy/webrunurl $base_s3/start/service production.ini paster

Deploy Web

#!/bin/bash -ex

wget -O- run.alestic.com/install/runurl | bashrunurl "https://${base_s3}/setup/add-keys-and-cert?SIGNATURE"runurl $base_s3/machine/apprunurl $base_s3/deploy/apprunurl $base_s3/start/service production.ini watchwoman_inotify watchwoman_pixels watchwoman_amqp spaceman \ crawlman billman generalman turtl quexecutor sphinx

Deploy App



DB Instance

• m2.2xlarge - 34GB RAM - 4 VCPU

• 8 EBS RAID 1+0

• PostgreSQL 8.4 log shipping to S3

• RAID snapshot every 6 hours

• Nightly SQL Backup



Generic Instance

• m1.large - 7.5 GB RAM - 2 VCPU

• 1 EBS with 500GB space

• Store frequently changed data

• Exact copies of each other



Web Instance

• c1.medium - 1.7 GB RAM - 2 VCPU

• No EBS

• Amazon ELB frontend

• Easily replaceable

• Logs aggregated separately in real time



AWS Tips/Tricks



runurlby Eric Hammond. Download and run any script from

any URL.



ec2-consistent-snapshotby Eric Hammond, atomic snapshot of EBS volumes

with xfs filesystem.



Eric HammondIf you want to know more about Amazon AWS,

he’s the one you want to talk to.



Use Public addressesPublic addresses are automatically resolved to internal

addresses when possible.



Use ELB and Cloudwatch

Very useful for latency and load monitoring… It also makes scaling the web frontend extremely easy!



EBS RAIDEBS is cheap. Instead of one big EBS create one big

soft-RAID/LVM volume from many of them, it’s faster and safer.



SMTPAuth / SendGridMail sending from Amazon EC2 was/is crippled. Use a

third party service to improve deliverability.



Use /etc/hostsSetting up a DNS is a lot of work at the beginning. Use SSH key names to setup a custom /etc/hosts file in each

instance.



new_lines = ["127.0.0.1 localhost"]for reservation in conn.get_all_instances(): for instance in reservation.instances: if instance.state == "running": KEYS.discard(instance.key_name) new_lines.append( "%s %s.internal %s" % ( socket.gethostbyname(instance.public_dns_name), instance.key_name, instance.key_name) )

for missing_key in KEYS: new_lines.append( "127.0.0.1 %s.internal %s" % (missing_key, missing_key) )

f = open('/etc/hosts.new', 'wb')f.write("\n".join(new_lines))f.write("\n")f.flush()f.close()shutil.move('/etc/hosts.new', '/etc/hosts')



AWS Wishlist



Elastic DNS ServiceDid I say that setting up your own DNS is hard?



Dedicated Rack Housing

Try to get a Netezza rack in AWS today...



Improve ELBIt’s a CNAME…

No HTTPS termination



Thank you!



Answers?



effective ec2

Technology