2014-08-27 | the twitter stack (vienna scala user group)
DESCRIPTION
Introduction to various open source projects by Twitter. Talk given at Vienna Scala User Group on Aug. 27, 2014.TRANSCRIPT
The Twitter StackFinagle, Ostrich, Zipkin,…
Dominik Gruber, @the_dom Vienna Scala User Group – Aug. 27, 2014
Dominik Gruber • @the_domTwitter Stack
Agenda• Introduction
• Prominent Adopters
• Finagle
• Ostrich
• Zipkin
• Iago
• Finatra
Dominik Gruber • @the_domTwitter Stack
Introduction
• Twitter started to move off Ruby on Rails in 2010
• Currently still on Scala 2.9
Dominik Gruber • @the_domTwitter Stack
Prominent Adopters• Box
• eBay
• Etsy
• Foursquare
• Gigya
• Nest
• Rdio
• SoundCloud
• Tumblr
Dominik Gruber • @the_domTwitter Stack
Dominik Gruber • @the_domTwitter Stack
Finagle“Finagle is an extensible RPC system for the
JVM, used to construct high-concurrency servers. Finagle implements uniform client
and server APIs for several protocols, and is designed for high performance and
concurrency. Most of Finagle’s code is protocol agnostic, simplifying the
implementation of new protocols.”
Dominik Gruber • @the_domTwitter Stack
Finagle provides…• Connection pools
• Failure detection
• Failover strategies
• Load balancers
• Back-pressure techniques
Dominik Gruber • @the_domTwitter Stack
Finagle…
• publishes standard statistics, logs, and exception reports
• supports distributed tracing across protocols
• optionally uses ZooKeeper for cluster management
• supports common sharding strategies
Dominik Gruber • @the_domTwitter Stack
Modules (Examples)• finagle-http
• finagle-memcached
• finagle-mysql
• finagle-protobuf
• finagle-redis
• finagle-thrift
Dominik Gruber • @the_domTwitter Stack
Basic abstractions
• Future
• Service
• Filter
Dominik Gruber • @the_domTwitter Stack
Ostrich
Dominik Gruber • @the_domTwitter Stack
Ostrich• A stats collector & reporter
• Collect runtime statistics (counters, gauges, metrics, and labels)
• Report those statistics through a simple web interface or into log files
• Similar to Metrics https://github.com/dropwizard/metrics
Dominik Gruber • @the_domTwitter Stack
Dominik Gruber • @the_domTwitter Stack
Zipkin
“Zipkin is a distributed tracing system that helps us gather timing data for all the
disparate services at Twitter. It manages both the collection and lookup of this data through
a Collector and a Query service.”
Dominik Gruber • @the_domTwitter Stack
Zipkin
Dominik Gruber • @the_domTwitter Stack
Zipkin’s Architecture
Dominik Gruber • @the_domTwitter Stack
Zipkin
• “Scribe” to transport traces to Zipkin and Hadoop
• Data stored in either Cassandra, Redis, HBase, MySQL, PostgreSQL, SQLite, or H2
Dominik Gruber • @the_domTwitter Stack
Iago
Dominik Gruber • @the_domTwitter Stack
Iago
• A load generator that replays production or synthetic traffic
• Iago supports arbitrarily high rates of traffic via built-in support for creating cluster
Dominik Gruber • @the_domTwitter Stack
Protocols• HTTP
• Thrift
• Memcache
• Kestrel (A queueing system from Twitter)
• UDP
Dominik Gruber • @the_domTwitter Stack
Finatra
Dominik Gruber • @the_domTwitter Stack
Dominik Gruber • @the_domTwitter Stack
Features• Fully Async using Futures
• Mustache Templates
• Detailed Metrics (p99,p999, etc)
• Profile/Debug over HTTP
• SSL/TLS Support
• Deploys with a single jar
Dominik Gruber • @the_domTwitter Stack
There’s more…
•
• Summingbird
• https://github.com/twitter
Dominik Gruber • @the_domTwitter Stack
Q & A
Dominik Gruber • @the_domTwitter Stack
Source• http://blog.oskarsson.nu/post/40196324612/the-
twitter-stack https://blog.twitter.com/2011/finagle-a-protocol-agnostic-rpc-system
• https://www.youtube.com/watch?v=whuT_-cFb5Ihttps://www.youtube.com/watch?v=QdER-Zl_YCM
• https://github.com/twitter https://github.com/finagle