sahara+storm: real time processing in sahara · 2019. 2. 26. · event stream processing tools...
Post on 22-Aug-2020
2 Views
Preview:
TRANSCRIPT
Sahara+Storm: Real time processing in Sahara
Presenters
2
Telles NóbregaSoftware Engineer - Universidade Federal de Campina Grande (Brazil)OpenStack ATC
Andrey BritoProfessor - Universidade Federal de Campina Grande (Brazil)
Michael McCuneSenior Software Engineer, Red Hat (USA)Sahara Core
Motivation● What is event stream processing?
○ Techniques for handling unbounded sequences of events○ Events are pieces of data carrying relevant information
○ Events are processed individually or in small batches (event windows)
● Why do we need it?○ Log/metric analysis, trending topics, fraud detection, IoT○ Value of information sometimes decreases quickly○ Sometimes data is too big to be stored in raw format
3
Motivation● Event Stream Processing Tools
○ Popular open-source tools (e.g., Apache Storm, Apache Samza, Apache Spark)○ Cloud-based (e.g., AWS Kinetic)
○ Academic (e.g., StreamMine3G)
● Storm○ Originated at BackType○ Acquired by Twitter○ Apache top-level project since 2014
4
Storm● Why Storm?
○ Well known and used○ Scalable○ Support for fault tolerance
○ Programing language agnostic
● Major components○ Nimbus○ Zookeeper○ Supervisor
Nimbus Zookeeper
Zookeeper
Zookeeper
Supervisor
Supervisor
Supervisor
Supervisor
Supervisor
5
Storm - Topology elements● Data into the system - Spouts● Data processing - Bolts
6https://storm.apache.org/
Storm - Communication between elements
Shuffle
Fields Global
All
7
Field = 1
Field = 2
Storm - Exclamation Topology
8
example example!!! example!!!!!!
https://github.com/apache/storm/tree/master/examples/storm-starter/
Storm - Exclamation Topology - Spout
9
Storm - Exclamation Topology - Spout
10
Storm - Exclamation Topology - Spout
11
Storm - Exclamation Topology
12
example example!!! example!!!!!!
https://github.com/apache/storm/tree/master/examples/storm-starter/
Storm - Exclamation Topology - Bolt
13
Storm - Exclamation Topology - Bolt
14
Storm - Exclamation Topology - Main
15
Storm - Exclamation Topology - Main
16
Storm - Exclamation Topology
● Building and Testing Locally
○ git clone https://github.com/apache/storm.git
○ mvn clean package
○ storm jar target/storm-starter-0.0.1-SNAPSHOT-standalone.jar storm.starter.ExclamationTopology exclamation-topology
17
Sahara Overview● OpenStack’s Data Processing service● Easy-to-use standard interfaces:
○ Provision cluster of machines○ Deploy data processing frameworks○ Scale clusters○ Run jobs on frameworks
● Full integration into OpenStack Dashboard● Support for a variety of processing frameworks
○ Hadoop, including vendor specific distributions○ Spark○ Storm
18
Sahara Architecture
19
Sahara● REST interface allows access to all features● Run jobs based of several types: PIG, Hive, Java, Shell, MapReduce and
Streaming● Access data stored in Swift, HDFS, or Manila● Elastic Data Processing with the ability to use static, transient, and scalable
clusters● Plugin based structure for data processing frameworks
○ Allows for addition of new frameworks
20
Sahara + Storm● Storm became part of Sahara in Kilo Release● Full UX of Storm via OpenStack
○ Horizon
21
Preparing a Cluster - Node Group Templates● Nimbus (Master)● Supervisor (Slave)● Zookeeper
22
Preparing a Cluster - Cluster Templates● Preparing your cluster
23
Preparing a Cluster - Starting ● Preparing your cluster
24
Preparing a Cluster - Storm UI
25
http://storm-master:8080
Running a Job - Binaries
26
Running a Job - Creating Template
27
Running a Job - Templates
28
Running a Job - Launch Job
29
Running a Job - Configs
30
Running a Job
31
Running a Job - Storm UI
32
storm-master:8080
Sahara + Storm - Next Steps● Improve integration with Data Sources
○ Kafka
○ Zaqar
● Add ability to run python jobs● Put newer versions of Storm into Sahara
○ 0.9.3○ 0.9.4○ 0.9.5
33
Demo
FilterSaharaTweets
34
FilterSaharaTweets
35
Wiring Topology
36
Demo
37
● Please Tweet!!!○ i.e.: Storm works great in #Sahara
Acknowledgements
38
Thank you!
Sahara + Storm
40
Sahara + Storm
41
Sahara + Storm
42
Sahara + Storm
43
AlarmBolt
44
Wiring Topology
45
top related