![Page 1: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/1.jpg)
Patterns for Deploying Analytics in the Real World
Sriskandarajah Suhothayan (Suho)Technical LeadWSO2
![Page 2: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/2.jpg)
What’s Analytics ?
![Page 3: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/3.jpg)
Problems to think about
• Can it handle my load ?• How costly it is ?• Adaptability ?• Can it analyse 3rd party systems ?• etc ...
![Page 4: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/4.jpg)
Where to start ?
![Page 5: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/5.jpg)
Where to start ?
• Think Big !
![Page 6: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/6.jpg)
Where to start ?
• Think Big !
But...
• Start simple !• Eat Your Own Dog Food • Analyse what you already have
![Page 7: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/7.jpg)
Step 1 :
Find Data Inside Your Organisation...
![Page 8: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/8.jpg)
Collect Data Internally
• Don’t worry about – Data formats – Data sources – Platforms – Protocols
Start with WSO2 DAS it has a unified data capturing framework !
![Page 9: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/9.jpg)
Deployment for Data Collection
![Page 10: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/10.jpg)
Step 2 :
Understand how things have been ...
![Page 11: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/11.jpg)
Deployment for Data AnalyticsBatch & Interactive Analytics
• Enable Searchability – Full text data– Drill down search
• See what has happened – Summarise the Data – Understand patterns and behaviors
![Page 12: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/12.jpg)
Deployment for Data AnalyticsBatch & Interactive Analytics
• Enable Searchability – Full text data– Drill down search
• See what has happened – Summarise the data – Understand patterns and behaviors
• Simple Deployment – 2 Nodes– Use RDBMS to store the data
![Page 13: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/13.jpg)
Deployment for Data AnalyticsBatch & Interactive Analytics
2 Node Deployment
![Page 14: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/14.jpg)
Step 3 :
Keep informed ...
![Page 15: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/15.jpg)
Deployment for Data AnalyticsRealtime Analytics
• Keep informed – Dashboard – Alerts– Feedback loops
![Page 16: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/16.jpg)
Deployment for Data AnalyticsRealtime Analytics
• Keep informed – Dashboard – Alerts– Feedback loops
• High Availability – Zero downtime– Zero data loss
![Page 17: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/17.jpg)
Realtime High Availability Deployment
Minimum 2 nodesMax throughput == 1 Node throughput
![Page 18: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/18.jpg)
Deployment for Data CommunicationAlerting & Communicating
Legacy & Internal Services
![Page 19: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/19.jpg)
Realtime + Batch Analytics
• Filter Data before you store – Realtime → Store & Process
• Summarize and store – Realtime → Store & Process
• Cross check with history – Lambda Architecture– Graph with Batch & Realtime
• Alerts based on batch processing – Batch → Realtime
From Batch
From Realtime
![Page 20: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/20.jpg)
Step 4 :
Think ahead ...
![Page 21: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/21.jpg)
Deployment for Data AnalyticsPredictive Analytics
1 Node of WSO2 ML 1 Node of WSO2 ML
![Page 22: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/22.jpg)
Minimum High Availability Deployment
All you need a 2 Node
Deployment
![Page 23: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/23.jpg)
Step 5 :
Expanding as a Connected Business …
![Page 24: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/24.jpg)
Deployment for Data Collection ...From 3rd Party Apps & Cloud
HTTP
Utilize API Analytics !
![Page 25: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/25.jpg)
Analyse Business with API Analytics
• APIs involved • Who invokes the APIs• Extract business information from
– Payloads– Resources URIs
Monetize APIs !
![Page 26: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/26.jpg)
Step 6 :
Scale with your Data ...
![Page 27: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/27.jpg)
Scaling Analytics Deployment The Changes !
• Realtime – Supported by Apache Storm
• For High Memory Requirement or CPU Intensive Processing – No query changes
• Batch – Move from RDBMS to HBase/Cassandra
• WSO2 DAS have a Data Abstraction Layer • Independent of underlying Data Store
Seamless migration :)
![Page 28: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/28.jpg)
Realtime Scalable Deployment ...
Event Processing offloaded to Siddhi Running on Apache Storm Seamlessly :)
![Page 29: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/29.jpg)
Realtime Scalable Deployment ...
Handling Stateless & Stateful Queries
![Page 30: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/30.jpg)
Realtime Scalable Deployment
Apache Storm Cluster + N CEP nodes
![Page 31: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/31.jpg)
Deployment for Scalable Data Analytics
Minimum 8 Nodes Deployment
(+ Storm if needed)
![Page 32: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/32.jpg)
Step 7 :
Sense the world around you ...
![Page 33: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/33.jpg)
Deployment for Data CollectionFrom Sensors
Analytics on the Edgewith WSO2 Siddhi
Push
![Page 34: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/34.jpg)
Deployment for Data CommunicationMobile & 3rd Party Apps
● Expose analytics results as API
○ Mobile Apps, Third Party
● Provides ○ Security, Billing, ○ Throttling, Quotas & SLA
● How ? ○ Write data to database from DAS ○ Build Services via WSO2 Data Services Server or use Analytics
REST API○ Expose them as APIs via WSO2 API Manager
![Page 35: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/35.jpg)
Analytics Life Cycle
Predefined analytics
• Artifacts bundled as CApps to and moved
Dev → Test → Preprod → Prod
Analytics on Production Environment
• Interactive Analytics• Personalizing Dashboards • Customised Alerts
![Page 36: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/36.jpg)
Summary
• Start small and scale as you grow• Minimum HA Deployment
– 2 Nodes
• Fully Distributed Deployment – 8+ Nodes – Scale based on need, horizontally and vertically
• Analyser, Indexer, Receiver, Realtime (With Apache Storm), Dashboard
![Page 37: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/37.jpg)
In God we trust; all others must bring data
- William Edwards Deming -
![Page 38: WSO2Con ASIA 2016: Patterns for Deploying Analytics in the Real World](https://reader031.vdocuments.us/reader031/viewer/2022021420/58a132241a28abd34f8b5917/html5/thumbnails/38.jpg)
Thank You