apache flink® meets apache mesos® and dc/os
TRANSCRIPT
Till Rohrmann [email protected] @stsffap
Apache Flink® Meets Apache Mesos® and DC/OS
Jörg Schad [email protected] @joerg_schad
MapReduce is crunching Data
We need to turn faster!
Evolution of Data Analytics
Batch Event ProcessingMicro-Batch
Days Hours Minutes Seconds Microseconds
Solves problems using predictive and prescriptive analytics
Reports what has happened using descriptive analytics
Predictive User InterfaceReal-time Pricing and Routing
Real-time Advertising
Billing,Chargeback
Product recommendations
FMACK Stack
EVENTS
Ubiquitous data streams from
connected devices
INGEST
Apache Kafka
STORE
Apache Flink
ANALYZE
Apache Cassandra
ACT
Akka
Ingest millions of events per second
Distributed & highly scalable database
Real-time and batch process data
Visualize data & build data driven apps
Mesos/ DC/OS
Sensors
Devices
Clients
Datacenter
Naive Approach
Typical Datacentersiloed, over-provisioned servers,
low utilization
Industry Average12-15% utilization
mySQL
microservice
Cassandra
Flink
Kafka
© 2017 Mesosphere, Inc. All Rights Reserved. 10
Apache Mesos
Typical Datacentersiloed, over-provisioned servers,
low utilization
Industry Average12-15% utilization
mySQL
microservice
Cassandra
Flink
Kafka
Mesosautomated schedulers, workload multiplexing
onto the same machines
Apache Mesos
Why Mesos?
! 2-level scheduling
! Fault-tolerant, battle-tested
! Scalable to 10,000+ nodes
! Created by Mesosphere founder @ UC Berkeley; used in production by 100+ web-scale companies [1]
[1] http://mesos.apache.org/documentation/latest/powered-by-mesos/
Apache Flink & Apache Mesos
Why Apache Mesos?▪ Mesos offers full functionality to implement fault
tolerant and elastic distributed applications
▪ 30% of survey respondents were running Flink on Mesos (prior to proper Mesos support, September 2016)
Flink’s Mesos Integration
Apache Flink FrameworkMesos Master
Mesos App Master
Flink MesosResourceManager
JobManager
Mesos Task
TaskManager
Mesos Task
TaskManager
Allocate Resources
Launch Mesos tasks
Register
Execute Job
Resource Manager Components
▪ Monitors connection to Mesos
Connection Monitor Launch Coordinator
▪ Resource offer processing and task scheduling
▪ Gathers offers and matches them to tasks using Fenzo
Task MonitorReconciliation Coordinator
▪ Monitors Mesos tasks ▪ Triggers reconciliation ▪ Makes sure tasks are properly killed
▪ Reconciles tasks view between ResourceManager and Mesos Master
Component Interplay
ResourceManager
Connection Monitor
Launch Coordinator
Task MonitorReconciliation Coordinator
Mesos MasterResource offers
Launch tasks
Monitor tasks
Status messages
Trigger reconciliation
Status messages
Mesos Task
Reconcile tasks
Start TaskManagers
Recover tasks
Kill task
Fenzo▪ Generic task scheduler for Mesos frameworks ▪ Developed by Netflix ▪ Matching between tasks and resource offers
▪ Pluggable fitness evaluator
Fenzo
Mesos
Launch Coordinator
Periodic resource offers
Tell Fenzo offered resources & tasks
Fenzo returns resource task matchings
Tasks to launch
New Distributed ArchitectureMesos Master
Mesos Cluster Client
(2) HTTP POST JobGraph/Jars
Flink Master ProcessFlink Mesos
ResourceManager
JobManager
(4) Start Process (and supervise)
(8) DeployTasks
(7) Register(5) Request slots
Flink Mesos Dispatcher
(3) Allocate containerfor Flink master
(6) Allocate containersfor TaskManagers
Marathon
(1) Start and monitor dispatcher
Mesos Task
TaskManager
Mesos Task
TaskManager
DC/OS
Datacenter Operating System (DC/OS)
Distributed Systems Kernel (Mesos)
Big Data + Analytics EnginesMicroservices (in containers)
Streaming
Batch
Machine Learning
Analytics
Functions & Logic
Search
Time Series
SQL / NoSQL
Databases
Modern App Components
Any Infrastructure (Physical, Virtual, Cloud)
Demo Time
Generator
▪ Financial data generated by generator ▪ Written to Kafka topics ▪ Kafka topics consumed by Flink ▪ Flink pipeline operates on Kafka data ▪ Results written back into Kafka
Conclusion
TL;DL
▪ Apache Flink runs on Mesos using Fenzo
▪ New distributed architecture supports dynamic resource allocation
▪ DC/OS offers easy to use Flink package
25
Thank you!@joerg_schad @stsffap @ApacheFlink @dataArtisans @dcos
We are hiring!
data-artisans.com/careers