kafka + uber- the world’s realtime transit infrastructure, aaron schildkrout
TRANSCRIPT
KAFKA +• Building the World's Realtime Transit Infrastructure
For Illustration only
SURGE - CIRCA 2013
SURGE - CIRCA 2016
DATA CONSUMERS
Real-time, Fast Analytics
BATCH PIPELINE
Storm
ApplicationsData Science
AnalyticsReporting
KAFKA
VERTICA
RIDER APP
DRIVER APP
API / SERVICES
DISPATCH (gps logs)
Mapping & Logistic Ad-hoc
exploration
ELK
Samza Alerts,Dashboards
Debugging
REAL-TIME PIPELINE
HADOOP
Surge Mobile App
DATA PRODUCERS
KAFKA 8 ECOSYSTEM @UBER
Product Features
Predictive Models
Operational Analytics
Business Intelligence
INFRASTRUCTURE ECOSYSTEM
NEAR REALTIME PRICE SURGING
PRODUCT FEATURES
FRAUD -ANOMALY DETECTION
PREDICTIVE MODELS
PREDICTIVE MODELS
ETA
OPERATIONAL ANALYTICS
UberEATs
OPERATIONAL ANALYTICS
XP
OPERATIONAL ANALYTICS
BUSINESS INTELLIGENCE
KAFKA 8KAFKA 7 MIGRATOR
Limited Availability
Difficult to Scale
Not multi-DC Multi-lang incompatibility Multi-DC, multi-languagesupport
2013
2014
2015 - 2016
KAFKA 7 WORLD
Difficult to Operate
Producer Scale Issues
High Availability
High Scalability
Kafka 7 + Mirrormaker
Deployed everywhere
Kafka 7 migratorDeployed
everywhere
New Kafka 8pipeline
Kafka 7
Mirrormaker
2.0
Rest architectu
re
Data AuditAutomatedTopic Mgmt
Logs Business events
Async REST library
Data Audit
Local spooling
High throughput custom protocol
REST ARCHITECTURE
Rest Proxy
Automated Schema and Topic Management
Mirrormaker 2.0
Robust
Data Audit
Dynamic topics
MIRROR MAKER 2.0
Destination DCSource DC
Msg counts across multiple DCs
End-end latencies across multiple DCs
DATA AUDIT FOR KAFKA MESSAGES
Mirrormaker
2.0
Rest architectu
re
Data Audit Kafka 8
AutomatedTopic Mgmt
A ROBUST FUTURE
0 data loss messaging systemData discovery and lineageQuota managementSelf-correcting brokersActive active data pipelines
Real-time Data
Dynamic SQL(ish)
Real-time decision
THE FUTURE
Real-time Data
Custom Application
Real-time decision
THE PRESENT
TELEMATICS
SELF DRIVING CAR
Thank you, Kafka Community!