Download - Start small bigger biggest
![Page 1: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/1.jpg)
Start Small Bigger BiggestThe Event Driven Organisation
Based on the Lambda Architecture
![Page 3: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/3.jpg)
Who Is That?
BigData.be
Belgian BigData Group
Founded in juli 2011
+200 members
http://www.bigdata.be
http://www.meetup.com/bigdatabe
@bigdatabe
![Page 4: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/4.jpg)
Agenda
TransCo
The Event Driven OrganisationEventsViewsIntegration
![Page 5: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/5.jpg)
TransCo
Meet TransCoInternational Transport
Track & TraceFreelance DriversCustom SoftwareEnterprise Service Bus
![Page 6: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/6.jpg)
TransCo
DepartmentsITFinanceLegalLogisticsSalesCommunicationsMarketing...
![Page 7: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/7.jpg)
TransCo
A customer requests a quote
A server goes down
A website visitor clicks on a link
A financial transaction is booked
A delivery truck pings its GPS coördinates
![Page 8: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/8.jpg)
TransCo
All these have a similar thing:
They Generate Events
![Page 9: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/9.jpg)
Events
ITFinanceLegalLogisticsSalesCommunications...
Events
![Page 10: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/10.jpg)
Events
Events used to manipulate their master data
![Page 11: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/11.jpg)
Events
Today, events ARE their master data
![Page 12: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/12.jpg)
TransCo
Isn't this a big data presentation?
Hang on, We'll get to that
![Page 13: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/13.jpg)
Events
An event isTime basedRecord linkedField orientedIncrementalImmutable
![Page 14: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/14.jpg)
Events
A new customer is addedADD [ts=1,id=123,field="name",value="Daan Gerits"]
ADD [ts=1,id=123,field="address",value="MyStreet 40"]
A customer is updatedADD [ts=13,id=123,field="address",value="AnotherStreet 1"]
Eventually a customer is removedADD [ts=43,id=123,field="address",value=]
![Page 15: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/15.jpg)
Events
Notice I only created new records
No UpdatesNo Deletes
CRUD becomes CR
![Page 16: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/16.jpg)
Events
Oh no, I shouldn't have done that!DELETE [ts=43,id=123,field="address"]
![Page 17: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/17.jpg)
TransCo
Discover market trends
Application and server statistics
Direct Marketing
Financial validation
Tracking Dashboard
![Page 18: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/18.jpg)
Views
View = Fx All Data
![Page 19: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/19.jpg)
Views
View =
![Page 20: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/20.jpg)
Different View TypesReal-timeRelationalReportingApplicationAPI...
Views
![Page 21: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/21.jpg)
Views areFull data modelsBased on all dataFocussed
Views
![Page 22: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/22.jpg)
Integration
How do I integrate with my existing infrastructure?
![Page 23: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/23.jpg)
Views can also be stored in- RDBMS- Indexing Engine- Files- DataWarehouse- ...
But these views are read-only!If you want read-write use the API
Integration
![Page 24: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/24.jpg)
Big Data
So what has all this to do with Big Data?
![Page 25: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/25.jpg)
Only appending eventsneed to store a lot of data
=> HDFS, HBase, ...
need batch processing=> MapReduce, Pig, Hive, ...
need real-time processing=> Storm, S4, ...
Big Data
![Page 26: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/26.jpg)
Allows trendingdriver statistics for his whole career
Allows state regenerationthe state of all master data on februari 12, 2005
Is human-error-proofremove the last event on this record
Scales very well
Event Driven Organisation
![Page 27: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/27.jpg)
By storing events
By generating focussed views
Event Driven Organisation
![Page 29: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/29.jpg)
Backup SlidesGenerating Views
![Page 30: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/30.jpg)
Views
Realtime Views
![Page 31: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/31.jpg)
Views
Latency
![Page 32: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/32.jpg)
Views
Generating takes time=> View is always x hours old
View Ready
Start Generating + + + ++
![Page 33: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/33.jpg)
Views
How to solve this?Store events in 2 locations
Master Data SetBuffer DB
Buffer DB has - data of last x hours- data, not events
![Page 34: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/34.jpg)
Views
Speed View
API
Batch View
![Page 35: Start small bigger biggest](https://reader034.vdocuments.us/reader034/viewer/2022052322/5590f1161a28ab50528b459c/html5/thumbnails/35.jpg)
Views
Batch ViewsUse MapReduce
Speed ViewsUse Storm or something similar