Sahara+Storm: Real time processing in Sahara

Presenters

2

Telles NóbregaSoftware Engineer - Universidade Federal de Campina Grande (Brazil)OpenStack ATC

Andrey BritoProfessor - Universidade Federal de Campina Grande (Brazil)

Michael McCuneSenior Software Engineer, Red Hat (USA)Sahara Core

Motivation● What is event stream processing?

○ Techniques for handling unbounded sequences of events○ Events are pieces of data carrying relevant information

○ Events are processed individually or in small batches (event windows)

● Why do we need it?○ Log/metric analysis, trending topics, fraud detection, IoT○ Value of information sometimes decreases quickly○ Sometimes data is too big to be stored in raw format

3

Motivation● Event Stream Processing Tools

○ Popular open-source tools (e.g., Apache Storm, Apache Samza, Apache Spark)○ Cloud-based (e.g., AWS Kinetic)

○ Academic (e.g., StreamMine3G)

● Storm○ Originated at BackType○ Acquired by Twitter○ Apache top-level project since 2014

4

Storm● Why Storm?

○ Well known and used○ Scalable○ Support for fault tolerance

○ Programing language agnostic

● Major components○ Nimbus○ Zookeeper○ Supervisor

Nimbus Zookeeper

Zookeeper

Zookeeper

Supervisor

Supervisor

Supervisor

Supervisor

Supervisor

5

Storm - Topology elements● Data into the system - Spouts● Data processing - Bolts

6https://storm.apache.org/

Storm - Communication between elements

Shuffle

Fields Global

All

7

Field = 1

Field = 2

Storm - Exclamation Topology

8

example example!!! example!!!!!!

https://github.com/apache/storm/tree/master/examples/storm-starter/

Storm - Exclamation Topology - Spout

9

Storm - Exclamation Topology - Spout

10

Storm - Exclamation Topology - Spout

11

Storm - Exclamation Topology

12

example example!!! example!!!!!!

https://github.com/apache/storm/tree/master/examples/storm-starter/

Storm - Exclamation Topology - Bolt

13

Storm - Exclamation Topology - Bolt

14

Storm - Exclamation Topology - Main

15

Storm - Exclamation Topology - Main

16

Storm - Exclamation Topology

● Building and Testing Locally

○ git clone https://github.com/apache/storm.git

○ mvn clean package

○ storm jar target/storm-starter-0.0.1-SNAPSHOT-standalone.jar storm.starter.ExclamationTopology exclamation-topology

17

Sahara Overview● OpenStack’s Data Processing service● Easy-to-use standard interfaces:

○ Provision cluster of machines○ Deploy data processing frameworks○ Scale clusters○ Run jobs on frameworks

● Full integration into OpenStack Dashboard● Support for a variety of processing frameworks

○ Hadoop, including vendor specific distributions○ Spark○ Storm

18

Sahara Architecture

19

Sahara● REST interface allows access to all features● Run jobs based of several types: PIG, Hive, Java, Shell, MapReduce and

Streaming● Access data stored in Swift, HDFS, or Manila● Elastic Data Processing with the ability to use static, transient, and scalable

clusters● Plugin based structure for data processing frameworks

○ Allows for addition of new frameworks

20

Sahara + Storm● Storm became part of Sahara in Kilo Release● Full UX of Storm via OpenStack

○ Horizon

21

Preparing a Cluster - Node Group Templates● Nimbus (Master)● Supervisor (Slave)● Zookeeper

22

Preparing a Cluster - Cluster Templates● Preparing your cluster

23

Preparing a Cluster - Starting ● Preparing your cluster

24

Preparing a Cluster - Storm UI

25

http://storm-master:8080

Running a Job - Binaries

26

Running a Job - Creating Template

27

Running a Job - Templates

28

Running a Job - Launch Job

29

Running a Job - Configs

30

Running a Job

31

Running a Job - Storm UI

32

storm-master:8080

Sahara + Storm - Next Steps● Improve integration with Data Sources

○ Kafka

○ Zaqar

● Add ability to run python jobs● Put newer versions of Storm into Sahara

○ 0.9.3○ 0.9.4○ 0.9.5

33

Demo

FilterSaharaTweets

34

FilterSaharaTweets

35

Wiring Topology

36

Demo

37

● Please Tweet!!!○ i.e.: Storm works great in #Sahara

Acknowledgements

38

Thank you!

Sahara + Storm

40

Sahara + Storm

41

Sahara + Storm

42

Sahara + Storm

43

AlarmBolt

44

Wiring Topology

45