(adv402) beating the speed of light with your infrastructure in aws | aws re:invent 2014
DESCRIPTION
With Amazon Web Services it's possible to serve the needs of modern high performance advertising without breaking the bank. This session covers how AdRoll processes more than 60 billion requests per day in less than 100 milliseconds each using Amazon DynamoDB, Auto Scaling, and Elastic Load Balancing. This process generates more than 2 GB of data every single second, which will be processed and turned into useful models over the following hour. We discuss designing systems that can answer billions of latency-sensitive global requests every day and look into some tricks to pare down the costs.TRANSCRIPT
November 14, 2014 | Las Vegas, NV
Valentino Volonghi, CTO, AdRoll
Siva Raghupathy, Principal Solutions Architect, AWS
60 billion requests/day
We
Must
Stay
Up
1% downtime
=
>$1M
No
Infinitely
Deep
Pockets
100ms MAX Latency
Paris-New York: ~6000km
Speed of Light in fiber: 200,000 km/s
RTT latency without hops and copper:
60ms
Paris-New York: ~6000km
Speed of Light in fiber: 200,000 km/s
RTT latency without hops and copper:
60ms6000
km
60 ms
c-RTT
a
a
a
r
• Memcache
aOpen source
aMature
aBlazingly fast
rNo strong guarantees
• Redis
aOpen source
rStorage scale
rNot really distributed
rOperationally intense.
• Hbase (we still use this)
aOpen source
aMaturing quickly
aGreat scale
rReally hard to operate
http://techblog.netflix.com/2014/07/revisiting-1-million-writes-per-second.html
Cassandra DynamoDB Delta
10/90 mix, $/month $287,064 $131,040 219%
50/50 mix, $/month $287,064 $280,800 ~0%
10/90, 3-yr reserved $27,075.6
($904k upfront)
$15,736
($504k upfront)
180%
• 10 people Cassandra ops team: $150k/month (fully loaded)
• 0 DynamoDB ops team: $0
Instance
Nginx Erlang Bidder Black Box
Supervisor
ASG
Amazon
Kinesis
Amazon S3
DynamoDB
Elastic Load
Balancing
Instance
Nginx Erlang Bidder Black Box
Supervisor
ASG
Elastic Load
Balancing
Amazon S3
Amazon
Kinesis
DynamoDB
DynamoDB
DynamoDB
HashKey Value
Key {‘segments’: set([‘123’, ‘345’]),
‘timestamp’: ‘1234556’,
‘leads’: ‘to-write-contention’}
HashKey RangeKey Value (just a single
timestamp)
Key Segment 1234554343254
Key Segment1 1231231433235
ArrayList
List WriteRequest ArrayList WriteRequest
Key
AtomicInteger AtomicInteger
Set WriteRequest CollectionsHashSet WriteRequest
String
AmazonDynamoDB
Runnable Runnable run
List WriteRequest
BatchWriteItemResult BatchWriteItemRequest
ImmutableMap List WriteRequest
Map
Not for me!
Data Collection
Data Storage
Global
Distribution
Bid Storage
Bidding
Data Collection
Data Storage
Global
Distribution
Bid Storage
Bidding
Data Collection = Batch Layer Bidding = Speed Layer
Data Collection
Data Storage
Global
Distribution
Bid Storage
Bidding
US East region
Availability Zone Availability Zone
Elastic Load
Balancing
instances instancesAuto Scaling group
Amazon S3Amazon
Kinesis
US East region
Availability Zone Availability Zone
Elastic Load
Balancing
instances instancesAuto Scaling group
Amazon S3Amazon
KinesisApache
StormDynamoDB
US West region
EU West region
DynamoDB
DynamoDB
BiddingData Collection
US East region
Availability Zone Availability Zone
Elastic Load Balancing
instances instances
Auto Scaling group
Amazon
S3
Amazon
KinesisApache
StormDynamoDB
Availability Zone Availability Zone
Auto Scaling group
Elastic Load Balancing
Data Collection
• Amazon EC2, Elastic Load Balancing, Auto Scaling
Store
• Amazon S3 + Amazon Kinesis
Global Distribution
• Apache Storm on Amazon EC2
Bid Store• DynamoDB
Bidding
• Amazon EC2, Elastic Load Balancing, Auto Scaling
Data Collection
Bidding
Ad Network 2Ad Network 1
Auto Scaling GroupAuto Scaling GroupAuto Scaling GroupAuto Scaling Group Auto Scaling GroupAuto Scaling Group
Auto Scaling GroupAuto Scaling Group Auto Scaling Group
Apache Storm
v2 V3 V3v1 v2 V3 V3v1
V2 V3 V3V1
Auto Scaling Group
V3 V4
Elastic Load Balancing Elastic Load Balancing Elastic Load Balancing Elastic Load Balancing
DynamoDB
Write
Read Read Read ReadRead Read
WriteWrites
WriteWrite
Read
V3`
Elastic Load Balancing
Elastic Load Balancing
Elastic Load Balancing
Elastic Load Balancing
Elastic Load Balancing
Elastic Load Balancing
DynamoDB
Data Collection
Bidding
DynamoDB
Write
Read
Read
Write
Write
WriteAmazon S3
Amazon Kinesis
