„enterprise event bus“ unified log (event) processing architecture
Post on 21-Jan-2017
1.166 Views
Preview:
TRANSCRIPT
2015 © Trivadis
BASEL BERN BRUGG LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN
2014 © Trivadis
„Enterprise Event Bus“Unified Log (Event) Processing ArchitectureDOAG DevCamp 2015
Guido Schmutz
Trivadis AG
April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture
1
2015 © Trivadis
A little story of a “real-life” customer situation
Traditional system interact with its clients and does its work
Implemented using legacy technologies (i.e. PL/SQL)
New requirement:
• Offer notification service to notify customer when goods are shipped
• Subscription and inform over different channels
• Existing technology doesn’t fit
May 2015Blueprints for the analysis of social media
2
delivery
LogisticSystem
Oracle
MobileApps
Sensor ship
sort
2
Rich(Web)ClientApps
DB
schedule
Logic(PL/SQL)
delivery
2015 © Trivadis
A little story of a “real-life” customer situation
May 2015Blueprints for the analysis of social media
3
delivery
LogisticSystem
Oracle
MobileApps
Sensor ship
sort
3
Rich(Web)ClientApps
DB
schedule
Notification
Logic(PL/SQL)
Logic(Java)delivery
SMS
…
Eventsare“owned”bytraditionalapplication(aswellasthechannelstheyaretransportedover)
2015 © Trivadis
A little story of a “real-life” customer situation
Rule Engine implemented in Java and invoked from OSB message flow
Notification system informed via queue
Higher Latency introduced (good enough in this case)
integrate in order to get the information!
Oracle Service Bus was already there
May 2015Blueprints for the analysis of social media
4
delivery
LogisticSystem
OracleOracle
ServiceBus
MobileApps
Sensor AQship
sort
4
Rich(Web)ClientApps
DB
schedule
Filter
Notification
Logic(PL/SQL)
JMS
RuleEngine(Java)
Logic(Java)delivery
shipdelivery
delivery true SMS
…
2015 © Trivadis
A little story of a “real-life” customer situation
May 2015Blueprints for the analysis of social media
5
delivery
LogisticSystem
OracleOracle
ServiceBus
MobileApps
Sensor AQship
sort
5
Rich(Web)ClientApps
DB
schedule
Filter
Notification
Logic(PL/SQL)
JMS
RuleEngine(Java)
Logic(Java)delivery
shipdelivery
delivery true SMS
…
Treat events as first-class citizens
Events belong to the “enterprise” and not an individual system => Catalog of Events similar to Catalog of Services/APIs !!
Event (stream) processing can be introduced and by that latency reduced!
2015 © Trivadis
Why Stream Processing?
Responselatency
StreamProcessing
Milliseconds tominutes
RPC
Synchronous Later.Possiblymuchlater.
April 2015Apache Storm vs. Spark Streaming - Two Stream Processing Platforms compared
6
2015 © Trivadis
Agenda
1. Designing Stream Processing Solutions
2. Implementing the Enterprise Event Bus (Unified Log)
3. Unified Log (Event) Processing Architecture in Action
April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture
7
2015 © Trivadis
What is Stream Processing?
April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture
8
EventStream
eventCollecting
event
Persist(Queue)
EventStream
eventCollecting
event
Processing
eventProcessing
result
result
EventStream
event Collecting/Processing
result
2015 © Trivadis
What is Stream Processing?
Infrastructure for continuous data processing
Computational model can be as general as MapReduce but with the ability to produce low-latency results
Data collected continuously is naturally processed continuously
aka. Event Processing / Complex Event Processing (CEP)
April 2015Apache Storm vs. Spark Streaming - Two Stream Processing Platforms compared
9
2015 © Trivadis
How to design a Streaming Processing System?It usually starts very simple … just one data pipeline
April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture
10
EventStream ConsumereventApplication
2015 © Trivadis
New Event Stream sources are added …
April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture
11
EventStream Consumer
2nd EventStream
3rd EventStream
nth EventStream
event
event
event
event
Application
2ndApplication
3rdApplication
Nth
Application
2015 © Trivadis
New Processors are interested in the events …
April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture
12
EventStream Consumer
2nd EventStream
3rd EventStream
nth EventStream
2nd Consumerevent
event
event
event
Application
2ndApplication
3rdApplication
Nth
Application
2015 © Trivadis
… and the solution becomes the problem
April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture
13
EventStream Consumer
2nd EventStream
3rd EventStream
nth EventStream
2nd Consumer
3rd Consumer
Nth Consumer
event
event
event
event
Application
2ndApplication
3rdApplication
Nth
Application
2015 © Trivadis
… and the solution becomes the problem
April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture
14
EventStream Consumer
2nd EventStream
3rd EventStream
nth EventStream
2nd Consumer
3rd Consumer
Nth Consumer
event
event
event
event
Application
2ndApplication
3rdApplication
Nth
Application
2015 © Trivadis
… and the solution becomes the problem
April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture
15
NewCustomers
OperationalLogs
ClickStream
MeterReadings
event
event
event
event
CDCCollector
LogCollector
ClickStreamCollector
SenorCollector
Hadoop/DataWarehouse
RecommendationSystem
LogSearch
FraudDetection
2015 © Trivadis
Decouple event streams from consumers
April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture
16
„UnifiedLog“
RememberEnterpriseServiceBus(ESB)?
EnterpriseEventBus EventStreamProcessorEventStreamSource
NewCustomers
OperationalLogs
ClickStream
MeterReadings
CDCCollector
LogCollector
ClickStreamCollector
SenorCollector
Hadoop/DataWarehouse
RecommendationSystem
LogSearch
FraudDetection
WhatistheideaofaUnifiedLog?
2015 © Trivadis
Unified Log – What is it?
By Unified Log, we do not mean this ….137.229.78.245 - - [02/Jul/2012:13:22:26 -0800] "GET /wp-admin/images/date-button.gif HTTP/1.1" 200 111137.229.78.245 - - [02/Jul/2012:13:22:26 -0800] "GET /wp-includes/js/tinymce/langs/wp-langs-en.js?ver=349-20805 HTTP/1.1" 200 13593137.229.78.245 - - [02/Jul/2012:13:22:26 -0800] "GET /wp-includes/js/tinymce/wp-tinymce.php?c=1&ver=349-20805 HTTP/1.1" 200 101114137.229.78.245 - - [02/Jul/2012:13:22:28 -0800] "POST /wp-admin/admin-ajax.php HTTP/1.1" 200 30747137.229.78.245 - - [02/Jul/2012:13:22:40 -0800] "POST /wp-admin/post.php HTTP/1.1" 302 -137.229.78.245 - - [02/Jul/2012:13:22:40 -0800] "GET /wp-admin/post.php?post=387&action=edit&message=1 HTTP/1.1" 200 73160137.229.78.245 - - [02/Jul/2012:13:22:41 -0800] "GET /wp-includes/css/editor.css?ver=3.4.1 HTTP/1.1" 304 -137.229.78.245 - - [02/Jul/2012:13:22:41 -0800] "GET /wp-includes/js/tinymce/langs/wp-langs-en.js?ver=349-20805 HTTP/1.1" 304 -137.229.78.245 - - [02/Jul/2012:13:22:41 -0800] "POST /wp-admin/admin-ajax.php HTTP/1.1" 200 30809
… but this
• a structured log (records are numbered beginning with 0 based on order they are written)
• aka. commit log orjournal
April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture
17
0 1 2 3 4 5 6 7 8 9 10 11
1st record Nextrecordwritten
2015 © Trivadis
Central Unified Log for (real-time) subscription
Take all the organization’s data (events) and put it into a central log for subscription
Properties of the Unified Log:• Unified: “Enterprise”, single deployment• Append-Only: events are appended, no update in place => immutable• Ordered: each event has an offset, which is unique within a shard• Fast: should be able to handle thousands of messages / sec• Distributed: lives on a cluster of machines
April 2015
„Enterprise Event Bus“ Unified Log (Event) Processing Architecture18
0 1 2 3 4 5 6 7 8 9 10 11
reads
writes
Collector
ConsumerSystemA(time=6)
ConsumerSystemB(time=10)
reads
2015 © Trivadis
Unified Log Processing Architecture
Stream processing allowsfor computing feedsoff other feeds
Derived feedsare no differentthan original feedsthey are computed off
Single deployment of“Unified Log”
logically different feeds
April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture
19
MeterReadings Collector
Enrich/Transform
AggregatebyMinute
Raw MeterReadings
Meter&Customer
Meterby CustomerbyMinute
Customer AggregatebyMinute
MeterbyMinute
Persist
MeterbyMinute
Persist
RawMeterReadings
….
2015 © Trivadis
Agenda
1. Designing Stream Processing Solutions
2. Implementing the Enterprise Event Bus (Unified Log)
3. Unified Log (Event) Processing Architecture in Action
April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture
20
2015 © Trivadis
Apache Kafka - Overview
• A distributed publish-subscribe messaging system
• Designed for processing of real time activity stream data (logs, metrics collections, social media streams, …)
• Initially developed at LinkedIn, now part of Apache
• Does not follow JMS Standards and does not use JMS API
• Kafka maintains feeds of messages in topics
April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture
21
Kafka Cluster
Consumer Consumer Consumer
Producer Producer Producer0 1 2 3 4 5 6 7 8 9 1
011
12
0 1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 10
11
12
Anatomy of a topic:
Partition 0
Partition 1
Partition 2
Writes
old new
2015 © Trivadis
Apache Kafka - Performance
Kafka at LinkedIn
Up to 2 million writes/sec on 3 cheap machines§ Using 3 producers on 3 different machines
April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture
22
10+billionwritesperday
172kmessagespersecond
(average)
55+billionmessagesperday
toreal-timeconsumers
http://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
2015 © Trivadis
Apache Kafka - Partition offsets
Offset: messages in the partitions are each assigned a unique (per partition) and sequential id called the offset
• Consumers track their pointers via (offset, partition, topic) tuples
April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture
23
ConsumergroupC1
2015 © Trivadis
Apache Kafka – two Options for Log Cleanup
April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture
24
1. Retaining a window of data• Ideal for event data• Window can be defined in time (days) or space (GBs) – defaults to 1 week
2. Retain a complete log (log compaction)• Ideal for keyed data• Keep a space-efficient complete
log of changes• Log compaction runs in the
background• Ensures that always at least the
last known value for each messagekey within the log of data is retained
2015 © Trivadis
Unified Log Alternatives
• Amazon Kinesis (http://aws.amazon.com/kinesis/)• Confluent (http://confluent.io/)• Redis Pub/Sub (http://redis.io/topics/pubsub)• Kestrel (http://robey.github.io/kestrel/)• ZeroMQ (http://zeromq.org/)• RabbitMQ (http://www.rabbitmq.com/)• Oracle GoldenGate (http://bit.ly/g-gate)• JMS compliant Server
• Apache ActiveMQ (http://activemq.apache.org/)• Weblogic JMS
(http://www.oracle.com/technetwork/middleware/weblogic/overview/index.html)
• IBM Websphere MQ (http://www-03.ibm.com/software/products/de/ibm-mq)• …
April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture
25
2015 © Trivadis
Agenda
1. Designing Stream Processing Solutions
2. Implementing the Enterprise Event Bus (Unified Log)
3. Unified Log (Event) Processing Architecture in Action
April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture
26
2015 © Trivadis
Unified Log Processing Architecture in Trivadis CRA
April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture
27
Tweets FilterandUnify
PersistTweet
FilteredTweets
SplitText
Words
CountoverTime
CountbyMinute
AnalyzeSentiment
Tweet&Sentiment
RemoveStopwords
Tweet
TweetsConsumer
TwitterFilterStream
SensorLayer DistributionLayer
SpeedLayer
Kafka Storm
Cassandra Elasticsearch
2015 © Trivadis
Unified Log Processing Architecture in Trivadis CRA
April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture
28
Tweets FilterandUnify
PersistTweet
FilteredTweets
SplitText
Words
CountoverTime
CountbyMinute
AnalyzeSentiment
Tweet&Sentiment
RemoveStopwords
Tweet
TweetsConsumer
TwitterFilterStream
SensorLayer DistributionLayer
Splitter
KafkaSpout
WordRemover
Splitter
WordRemover
Shuffle Fields
Kafka
Kafka
WordRemover
StormTopology
SpeedLayer
Kafka Storm
Cassandra Elasticsearch
2015 © Trivadis
Weitere Informationen...
April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture
29
2015 © Trivadis
BASEL BERN BRUGG LAUSANNE ZÜRICH DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. HAMBURG MÜNCHEN STUTTGART WIEN
Fragen und Antworten...
2013 © Trivadis
April 2015„Enterprise Event Bus“ Unified Log (Event) Processing Architecture
top related