q con shanghai2013-[jains krums]-[real-time-delivery-archiecture]
TRANSCRIPT
@raffi qcon shanghai 2013
real-time delivery architecture
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
http://twitpic.com/135xa - There’s a plane in the Hudson. I’m on a ferry going to pick up the people. Crazy.!15 Jan 09 Janis Krums @jkrums
@raffi qcon shanghai 2013
what are the goals?⇢evolve from being solely a web stack
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
LOGICPRESENTATIONROUTING
T-Bird
T-Flock + Haplo
Flock(s)
Gizmoduck
Monorail
STORAGE & RETRIEVAL
@raffi qcon shanghai 2013
PRESENTATIONROUTING
T-Bird
T-Flock + Haplo
Flock(s)
Gizmoduck
Monorail
STORAGE & RETRIEVAL
LOGIC
@raffi qcon shanghai 2013
ROUTING
T-Bird
T-Flock + Haplo
Flock(s)
Gizmoduck
Monorail
STORAGE & RETRIEVAL
LOGICPRESENTATION
@raffi qcon shanghai 2013
T-Bird
T-Flock + Haplo
Flock(s)
Gizmoduck
Monorail
STORAGE & RETRIEVAL
LOGICPRESENTATIONROUTING
@raffi qcon shanghai 2013
T-Flock + Haplo
Flock(s)
Gizmoduck
Monorail
STORAGE & RETRIEVAL
LOGICPRESENTATIONROUTING
T-Bird
@raffi qcon shanghai 2013
Flock(s)
Gizmoduck
Monorail
STORAGE & RETRIEVAL
LOGICPRESENTATIONROUTING
T-Bird
T-Flock + Haplo
@raffi qcon shanghai 2013
Flock(s)
Monorail
STORAGE & RETRIEVAL
LOGICPRESENTATIONROUTING
T-Bird
T-Flock + Haplo
Darkwing
@raffi qcon shanghai 2013
Monorail
STORAGE & RETRIEVAL
LOGICPRESENTATIONROUTING
T-Bird
T-Flock + Haplo
Flock(s)
Darkwing
@raffi qcon shanghai 2013
STORAGE & RETRIEVAL
LOGICPRESENTATIONROUTING
Monorail
T-Bird
T-Flock + Haplo
Flock(s)
Darkwing
@raffi qcon shanghai 2013
STORAGE & RETRIEVAL
LOGICPRESENTATIONROUTING
Monorail
T-Bird
T-Flock + Haplo
Flock(s)
Darkwing
@raffi qcon shanghai 2013
what are the goals?⇢evolve from being solely a web stack
⇢ isolate responsibilities and concerns
⇢site speed and reliability
⇢developer innovation speed
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Pull Push
Targeted twitter.com home_timeline API
User / Site Streams Mobile Push (SMS, etc.)
Queried Search API Track / Follow Streams
@raffi qcon shanghai 2013
Pull Push
Targeted twitter.com home_timeline API
User / Site Streams Mobile Push (SMS, etc.)
Queried Search API Track / Follow Streams
@raffi qcon shanghai 2013
Pull Push
Targeted twitter.com home_timeline API
User / Site Streams Mobile Push (SMS, etc.)
Queried Search API Track / Follow Streams
@raffi qcon shanghai 2013
Pull Push
Targeted twitter.com home_timeline API
User / Site Streams Mobile Push (SMS, etc.)
Queried Search API Track / Follow Streams
@raffi qcon shanghai 2013
Pull Push
Targeted twitter.com home_timeline API
User / Site Streams Mobile Push (SMS, etc.)
Queried Search API Track / Follow Streams
@raffi qcon shanghai 2013
Pull Push
Targeted twitter.com home_timeline API
User / Site Streams Mobile Push (SMS, etc.)
Queried Search API Track / Follow Streams
@raffi qcon shanghai 2013
Pull Push
Targeted twitter.com home_timeline API
User / Site Streams Mobile Push (SMS, etc.)
Queried Search API Track / Follow Streams
@raffi qcon shanghai 2013
Pull Push
Targeted twitter.com home_timeline API
User / Site Streams Mobile Push (SMS, etc.)
Queried Search API Track / Follow Streams
@raffi qcon shanghai 2013
Pull Push
Targeted twitter.com home_timeline API
User / Site Streams Mobile Push (SMS, etc.)
Queried Search API Track / Follow Streams
@raffi qcon shanghai 2013
Pull Push
Targeted twitter.com home_timeline API
User / Site Streams Mobile Push (SMS, etc.)
Queried Search API Track / Follow Streams
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Write API
Fanout
Tim
elin
e C
ache
Redis
Ingester
Sear
ch C
ache
Redis
Push
Com
pute
HTTP Push
Bat
ch C
ompu
te
Hadoop
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Ingester
Write API
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Write API
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Tim
elin
e C
ache
RedisRedis
Redis
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Ingester
Sear
ch C
ache
Push
Com
pute
Bat
ch C
ompu
te
Hadoop
Write API
Fanout
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
RedisRedis
Redis
Timeline Service
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
Tim
elin
e C
ache
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
RedisRedis
Timeline Service
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
Redis
Tim
elin
e C
ache
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
RedisRedis
Timeline Service
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
Redis
Tim
elin
e C
ache
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Timeline Service
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Redis
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Timeline Service
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Redis
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Timeline Service
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Redis
Social Graph Service
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Timeline Service
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Redis
Social Graph Service
insert⇢keyed off
“recipient”
⇢pipelined 4k “destinations” at a time
⇢replicated @raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Timeline Service
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Redis
using redis⇢native list
structure
⇢RPUSHX to only add to cached timelines
Tweet ID BitsUser ID
8 bytes 4 bytes8 bytes
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Timeline Service
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Redis
using redis⇢native list
structure
⇢RPUSHX to only add to cached timelines
Tweet ID BitsUser ID
Tweet ID BitsUser ID
Tweet ID BitsUser ID
Tweet ID BitsUser ID
Tweet ID BitsUser ID
Tweet ID BitsUser ID
Tweet ID BitsUser ID
Tweet ID BitsUser ID
Tweet ID BitsUser ID
Tweet ID BitsUser ID
@raffi qcon shanghai 2013
Timeline Service
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Redis
using redis⇢native list
structure
⇢RPUSHX to only add to cached timelines
Tweet ID BitsUser ID
Tweet ID BitsUser ID
Tweet ID BitsUser ID
Tweet ID BitsUser ID
Tweet ID BitsUser ID
Tweet ID BitsUser ID
Tweet ID BitsUser ID
Tweet ID BitsUser ID
Tweet ID BitsUser ID
Tweet ID BitsUser ID Tweet ID
Tweet ID
Tweet ID
@raffi qcon shanghai 2013
Timeline Service
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Redis
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Timeline Service
Redis
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Timeline Service
Redis
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Timeline Service
Redis
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Pull Push
Targeted twitter.com home_timeline API
User / Site Streams Mobile Push (SMS, etc.)
Queried Search API Track / Follow Streams
@raffi qcon shanghai 2013
Ingester
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Timeline Service
Redis
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Timeline Service
Ingester
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Redis
@raffi qcon shanghai 2013
Sear
ch C
ache
RedisRedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Timeline Service
Ingester
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Redis
@raffi qcon shanghai 2013
EarlybirdRedis
Earlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Timeline Service
Ingester
Sear
ch In
dex
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Redis
@raffi qcon shanghai 2013
RedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Timeline Service
Ingester
Sear
ch In
dex
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Redis
@raffi qcon shanghai 2013
RedisEarlybird
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Timeline Service
Ingester
Sear
ch In
dex
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
Redis
@raffi qcon shanghai 2013
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
HadoopSe
arch
Inde
xRedis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybird
@raffi qcon shanghai 2013
Blender
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
HadoopSe
arch
Inde
xRedis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybird
@raffi qcon shanghai 2013
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
HadoopSe
arch
Inde
x
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybird
@raffi qcon shanghai 2013
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
HadoopSe
arch
Inde
x
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybird
blender⇢queries one
replica of all indexes
⇢merges & ranks results
@raffi qcon shanghai 2013
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
HadoopSe
arch
Inde
x
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybird
@raffi qcon shanghai 2013
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
HadoopSe
arch
Inde
x
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybird
@raffi qcon shanghai 2013
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
HadoopSe
arch
Inde
x
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybird
@raffi qcon shanghai 2013
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
HadoopSe
arch
Inde
x
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybird
@raffi qcon shanghai 2013
Pull Push
Targeted twitter.com home_timeline API
User / Site Streams Mobile Push (SMS, etc.)
Queried Search API Track / Follow Streams
@raffi qcon shanghai 2013
Push
Com
pute
HTTP Push
Mobile Push B
atch
Com
pute
HadoopSe
arch
Inde
x
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybird
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
HTTP Push
Mobile Push B
atch
Com
pute
Hadoop
Push
Com
pute
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybird
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Mobile Push B
atch
Com
pute
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybird
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Mobile Push B
atch
Com
pute
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybird
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
⇢maintains persistent connections with end clients
⇢processes tweet & social graph events
⇢event-based “router”
http push / hosebird
@raffi qcon shanghai 2013
Hosebird
Hosebird
Hosebird Firehose
User Streams
Track / Follow
Write API
Text
@raffi qcon shanghai 2013
Hosebird
Hosebird Firehose
User Streams
Track / Follow
HosebirdWrite API
@raffi qcon shanghai 2013
Hosebird
Hosebird Firehose
User Streams
Track / Follow
HosebirdWrite API
event propagation⇢write API sends all events into hosebird;
sees content creation events, social graph changes, etc.
⇢different queues for public tweets, protected tweets, social events, etc.
@raffi qcon shanghai 2013
Firehose
User Streams
Track / Follow
Hosebird
Hosebird
Hosebird
Write API
@raffi qcon shanghai 2013
Firehose
User Streams
Track / Follow
Hosebird
Hosebird
Hosebird
Write API
event cascading⇢bandwidth management
⇢simultaneous connection management (~1m long lived & open connections to this cluster)
@raffi qcon shanghai 2013
User Streams
Track / Follow
Firehose
Hosebird
Hosebird
Hosebird
Write API
@raffi qcon shanghai 2013
User Streams
Track / Follow
Firehose
Hosebird
Hosebird
Hosebird
Write API
firehose⇢edge machine simply outputs the public
tweet queue
⇢only allow a limited number of firehoses per hosebird box for bandwidth management
@raffi qcon shanghai 2013
User Streams
Firehose
Track / FollowHosebird
Hosebird
Hosebird
Write API
@raffi qcon shanghai 2013
User Streams
Firehose
Track / FollowHosebird
Hosebird
Hosebird
Write API
track / follow⇢simple query based on tweet content
⇢keeps list of terms / users of interest
⇢parses public tweets at the edge, and if term matches a token, or user is of interest, then route
@raffi qcon shanghai 2013
Firehose
Track / Follow
User Streams
Hosebird
Hosebird
Hosebird
Write API
@raffi qcon shanghai 2013
Firehose
Track / Follow
User Streams
Hosebird
Hosebird
Hosebird
Write API
user streams⇢replicate home timeline experience
⇢upon login, obtain “following” list
⇢keep cached following list coherent by seeing social graph updates
⇢route tweet if from a followed user
@raffi qcon shanghai 2013
Mobile Push B
atch
Com
pute
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybird
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Mobile Push B
atch
Com
pute
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybird
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Mobile Push B
atch
Com
pute
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybird
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Pull Push
Targeted twitter.com home_timeline API
User / Site Streams Mobile Push (SMS, etc.)
Queried Search API Track / Follow Streams
@raffi qcon shanghai 2013
Mobile Push B
atch
Com
pute
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybird
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Bat
ch C
ompu
te
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Bat
ch C
ompu
te
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
@raffi qcon shanghai 2013
Bat
ch C
ompu
te
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
Social Graph Service
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Bat
ch C
ompu
te
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
Social Graph Service
@raffi qcon shanghai 2013
Bat
ch C
ompu
te
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
Social Graph Service
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Bat
ch C
ompu
te
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
Social Graph Service
@raffi qcon shanghai 2013
Bat
ch C
ompu
te
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Pull Push
Targeted twitter.com home_timeline API
User / Site Streams Mobile Push (SMS, etc.)
Queried Search API Track / Follow Streams
@raffi qcon shanghai 2013
Bat
ch C
ompu
te
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Hadoop
Bat
ch C
ompu
te
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Bat
ch C
ompu
te
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Bat
ch C
ompu
te
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
@raffi qcon shanghai 2013
Bat
ch C
ompu
te
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Bat
ch C
ompu
te
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Bat
ch C
ompu
te
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Asynchronous Path
Query Path
Bat
ch C
ompu
te
Hadoop
Synchronous Path
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Synchronous Path
Query Path
Bat
ch C
ompu
te
Hadoop
Asynchronous Path
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Asynchronous Path
Synchronous Path
Bat
ch C
ompu
te
Hadoop
Query Path
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Bat
ch C
ompu
te
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Writ
e Pa
th
Bat
ch C
ompu
te
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
Rea
d Pa
th
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Rea
d Pa
th
Bat
ch C
ompu
te
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
Writ
e Pa
th
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Bat
ch C
ompu
te
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
things we’re trying...
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Write API
Ingester Fanout
Sear
ch In
dex
RedisEarlybird
EarlybirdRedis
RedisRedis
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Ingester Fanout
Tim
elin
e C
ache
Sear
ch In
dex
RedisRedis
RedisRedisEarlybird
Earlybird
Write API
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Ingester Fanout
Tim
elin
e C
ache
Sear
ch In
dex
RedisRedis
RedisRedisEarlybird
Earlybird
Write API
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Tim
elin
e C
ache
Sear
ch In
dex
RedisRedis
RedisRedisEarlybird
Earlybird
Write API
Ingester Fanout
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Tim
elin
e C
ache
Sear
ch In
dex
RedisRedis
RedisRedisEarlybird
Earlybird
Write API
Ingester Fanout
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
RedisRedis
RedisRedisEarlybird
Earlybird
Write API
Ingester Fanout
Sear
ch In
dex
Tim
elin
e C
ache
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Write API
Ingester Fanout
Sear
ch In
dex
RedisEarlybird
EarlybirdRedis
RedisRedis
Tim
elin
e C
ache
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
Write API
Ingester Fanout
Sear
ch In
dex
RedisEarlybird
EarlybirdRedis
RedisRedis
Tim
elin
e C
ache
search index ⇢[‘hello’,‘world’]
fanout index ⇢[@danadanger, ...]
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
User Intent Query Expansion
“Hello, world” “Hello” AND “world”
@raffi’s home timeline home_timeline:raffi
@raffi qcon shanghai 2013
User Intent Query Expansion
“Hello, world” “Hello” AND “world”
@raffi’s home timeline home_timeline:raffi
@raffi qcon shanghai 2013
User Intent Query Expansion
“Hello, world” “Hello” AND “world”
@raffi’s home timeline home_timeline:raffi
@raffi qcon shanghai 2013
User Intent Query Expansion
“Hello, world” “Hello” AND “world”
@raffi’s home timeline home_timeline:raffi
@raffi qcon shanghai 2013
User Intent Query Expansion
“Hello, world” “Hello” AND “world”
@raffi’s home timeline home_timeline:raffi
@raffi qcon shanghai 2013
User Intent Query Expansion
“Hello, world” “Hello” AND “world”
@raffi’s home timeline home_timeline:raffi
@raffi qcon shanghai 2013
User Intent Query Expansion
“Hello, world” “Hello” AND “world”
@raffi’s home timeline home_timeline:raffi
@raffi qcon shanghai 2013
User Intent Query Expansion
“Hello, world” “Hello” AND “world”
@raffi’s home timeline home_timeline:raffi
@raffi qcon shanghai 2013
Write API
Ingester Fanout
Sear
ch In
dex
RedisEarlybird
EarlybirdRedis
RedisRedis
Tim
elin
e C
ache
fan-in⇢O(1) write
⇢O(n) read
fan-out⇢O(n) write
⇢O(1) read
@raffi qcon shanghai 2013
User Intent Query Expansion
“Hello, world” “Hello” AND “world”
@raffi’s home timeline home_timeline:raffi
@raffi qcon shanghai 2013
User Intent Query Expansion
“Hello, world” “Hello” AND “world”
@raffi’s home timelinehome_timeline:raffi
OR user_timeline:taylorswift13
@raffi qcon shanghai 2013
streaming compute⇢continuous computation
⇢driven by the events that come into twitter
⇢generalizing the push mechanism
@raffi qcon shanghai 2013
Bat
ch C
ompu
te
Hadoop
Push
Com
pute
HTTP Push
Sear
ch In
dex
Blender
Redis
Timeline Service
Ingester
Earlybird
Write API
Fanout
RedisRedis
Tim
elin
e C
ache
RedisEarlybirdMobile Push
@raffi qcon shanghai 2013
timeline query statistics⇢>150m active users worldwide
⇢300k qps poll-based timelines @ 1ms p50 / 4ms p99
⇢30k qps search-based timelines
@raffi qcon shanghai 2013
tweet input⇢~340m tweets per day
⇢~4K/sec daily average
⇢~6K/sec daily peak
⇢>10K/sec during large events
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
followingfollowed by
@raffi qcon shanghai 2013
@raffi qcon shanghai 2013
timeline delivery statistics⇢26b deliveries / day (~18m / min)
⇢3.5 seconds @ p50 to deliver to 1m
⇢~300k deliveries / sec
thanks!